fastlane Running xcrun extractLocStrings as part of Fastlane causes crash for duplicated high-codepoint unicode localisable strings - Ruby

New Issue Checklist

Issue Description

We run xcrun extractLocStrings as part of our fastlane workflow and we have a project where some of the localisable strings contain high-codepoint unicode characters such as "µm" where the µ is the high-codepoint character. This string is localisable in multiple parts of the codebase and by default xcrun extractLocStrings outputs standard output warning you that you have used multiple comments for those strings.

When fastlane outputs to screen the first instance of one of the warnings where the localisable string contains a high-codepoint character, it crashes with the stacktrace shown further below.

I was able to supress the output of the warnings from xcrun extractLocStrings by using the flag -q but this is not the default and fastlane should be able to handle the output generated by any tool.

Command executed

bin/fastlane test

Complete output when running fastlane, including the stack trace and command used
/Users/my_user/buildAgent/work/1234567890abcdef/_vendor/bundle/ruby/2.3.0/gems/fastlane-2.127.0/fastlane_core/lib/fastlane_core/ui/implementations/shell.rb:71:in `split': [!] invalid byte sequence in UTF-8 (ArgumentError)
    from /Users/my_user/buildAgent/work/1234567890abcdef/_vendor/bundle/ruby/2.3.0/gems/fastlane-2.127.0/fastlane_core/lib/fastlane_core/ui/implementations/shell.rb:71:in `command_output'
    from /Users/my_user/buildAgent/work/1234567890abcdef/_vendor/bundle/ruby/2.3.0/gems/fastlane-2.127.0/fastlane_core/lib/fastlane_core/ui/ui.rb:17:in `method_missing'
    from /Users/my_user/buildAgent/work/1234567890abcdef/_vendor/bundle/ruby/2.3.0/gems/fastlane-2.127.0/fastlane/lib/fastlane/helper/sh_helper.rb:57:in `block (2 levels) in sh_control_output'
    from /Users/my_user/buildAgent/work/1234567890abcdef/_vendor/bundle/ruby/2.3.0/gems/fastlane-2.127.0/fastlane/lib/fastlane/helper/sh_helper.rb:56:in `each'
    from /Users/my_user/buildAgent/work/1234567890abcdef/_vendor/bundle/ruby/2.3.0/gems/fastlane-2.127.0/fastlane/lib/fastlane/helper/sh_helper.rb:56:in `block in sh_control_output'
    from /System/Library/Frameworks/Ruby.framework/Versions/2.3/usr/lib/ruby/2.3.0/open3.rb:205:in `popen_run'
    from /System/Library/Frameworks/Ruby.framework/Versions/2.3/usr/lib/ruby/2.3.0/open3.rb:190:in `popen2e'
    from /Users/my_user/buildAgent/work/1234567890abcdef/_vendor/bundle/ruby/2.3.0/gems/fastlane-2.127.0/fastlane/lib/fastlane/helper/sh_helper.rb:54:in `sh_control_output'
    from /Users/my_user/buildAgent/work/1234567890abcdef/_vendor/bundle/ruby/2.3.0/gems/fastlane-2.127.0/fastlane/lib/fastlane/helper/sh_helper.rb:16:in `sh_no_action'
    from /Users/my_user/buildAgent/work/1234567890abcdef/_vendor/bundle/ruby/2.3.0/gems/fastlane-2.127.0/fastlane/lib/fastlane/fast_file.rb:213:in `block in sh'
    from /Users/my_user/buildAgent/work/1234567890abcdef/_vendor/bundle/ruby/2.3.0/gems/fastlane-2.127.0/fastlane/lib/fastlane/actions/actions_helper.rb:50:in `execute_action'
    from /Users/my_user/buildAgent/work/1234567890abcdef/_vendor/bundle/ruby/2.3.0/gems/fastlane-2.127.0/fastlane/lib/fastlane/fast_file.rb:212:in `sh'
    from /Users/my_user/buildAgent/work/1234567890abcdef/_vendor/bundle/ruby/2.3.0/gems/fastlane-2.127.0/fastlane/lib/fastlane/fast_file.rb:206:in `sh'
    from Fastfile:145:in `block (2 levels) in parsing_binding'
    from /Users/my_user/buildAgent/work/1234567890abcdef/_vendor/bundle/ruby/2.3.0/gems/fastlane-2.127.0/fastlane/lib/fastlane/lane.rb:33:in `call'
    from /Users/my_user/buildAgent/work/1234567890abcdef/_vendor/bundle/ruby/2.3.0/gems/fastlane-2.127.0/fastlane/lib/fastlane/runner.rb:204:in `try_switch_to_lane'
    from /Users/my_user/buildAgent/work/1234567890abcdef/_vendor/bundle/ruby/2.3.0/gems/fastlane-2.127.0/fastlane/lib/fastlane/runner.rb:146:in `trigger_action_by_name'
    from /Users/my_user/buildAgent/work/1234567890abcdef/_vendor/bundle/ruby/2.3.0/gems/fastlane-2.127.0/fastlane/lib/fastlane/fast_file.rb:154:in `method_missing'
    from Fastfile:176:in `block (2 levels) in parsing_binding'
    from /Users/my_user/buildAgent/work/1234567890abcdef/_vendor/bundle/ruby/2.3.0/gems/fastlane-2.127.0/fastlane/lib/fastlane/lane.rb:33:in `call'
    from /Users/my_user/buildAgent/work/1234567890abcdef/_vendor/bundle/ruby/2.3.0/gems/fastlane-2.127.0/fastlane/lib/fastlane/runner.rb:49:in `block in execute'
    from /Users/my_user/buildAgent/work/1234567890abcdef/_vendor/bundle/ruby/2.3.0/gems/fastlane-2.127.0/fastlane/lib/fastlane/runner.rb:45:in `chdir'
    from /Users/my_user/buildAgent/work/1234567890abcdef/_vendor/bundle/ruby/2.3.0/gems/fastlane-2.127.0/fastlane/lib/fastlane/runner.rb:45:in `execute'
    from /Users/my_user/buildAgent/work/1234567890abcdef/_vendor/bundle/ruby/2.3.0/gems/fastlane-2.127.0/fastlane/lib/fastlane/lane_manager.rb:56:in `cruise_lane'
    from /Users/my_user/buildAgent/work/1234567890abcdef/_vendor/bundle/ruby/2.3.0/gems/fastlane-2.127.0/fastlane/lib/fastlane/command_line_handler.rb:36:in `handle'
    from /Users/my_user/buildAgent/work/1234567890abcdef/_vendor/bundle/ruby/2.3.0/gems/fastlane-2.127.0/fastlane/lib/fastlane/commands_generator.rb:108:in `block (2 levels) in run'
    from /Users/my_user/buildAgent/work/1234567890abcdef/_vendor/bundle/ruby/2.3.0/gems/commander-fastlane-4.4.6/lib/commander/command.rb:178:in `call'
    from /Users/my_user/buildAgent/work/1234567890abcdef/_vendor/bundle/ruby/2.3.0/gems/commander-fastlane-4.4.6/lib/commander/command.rb:153:in `run'
    from /Users/my_user/buildAgent/work/1234567890abcdef/_vendor/bundle/ruby/2.3.0/gems/commander-fastlane-4.4.6/lib/commander/runner.rb:476:in `run_active_command'
    from /Users/my_user/buildAgent/work/1234567890abcdef/_vendor/bundle/ruby/2.3.0/gems/fastlane-2.127.0/fastlane_core/lib/fastlane_core/ui/fastlane_runner.rb:76:in `run!'
    from /Users/my_user/buildAgent/work/1234567890abcdef/_vendor/bundle/ruby/2.3.0/gems/commander-fastlane-4.4.6/lib/commander/delegates.rb:15:in `run!'
    from /Users/my_user/buildAgent/work/1234567890abcdef/_vendor/bundle/ruby/2.3.0/gems/fastlane-2.127.0/fastlane/lib/fastlane/commands_generator.rb:333:in `run'
    from /Users/my_user/buildAgent/work/1234567890abcdef/_vendor/bundle/ruby/2.3.0/gems/fastlane-2.127.0/fastlane/lib/fastlane/commands_generator.rb:41:in `start'
    from /Users/my_user/buildAgent/work/1234567890abcdef/_vendor/bundle/ruby/2.3.0/gems/fastlane-2.127.0/fastlane/lib/fastlane/cli_tools_distributor.rb:119:in `take_off'
    from /Users/my_user/buildAgent/work/1234567890abcdef/_vendor/bundle/ruby/2.3.0/gems/fastlane-2.127.0/bin/fastlane:23:in `<top (required)>'
    from /Users/my_user/buildAgent/work/1234567890abcdef/bin/fastlane:29:in `load'
    from /Users/my_user/buildAgent/work/1234567890abcdef/bin/fastlane:29:in `<main>'

Environment

✅ fastlane environment ✅

Stack

Key Value
OS 10.14.6
Ruby 2.3.7
Bundler? true
Git git version 2.20.1 (Apple Git-117)
Installation Source bin/fastlane
Host Mac OS X 10.14.6 (18G87)
Ruby Lib Dir /System/Library/Frameworks/Ruby.framework/Versions/2.3/usr/lib
OpenSSL Version LibreSSL 2.6.5
Is contained false
Is homebrew false
Is installed via Fabric.app false
Xcode Path /Applications/Xcode1030.app/Contents/Developer/
Xcode Version 10.3

System Locale

Variable Value
LANG en_GB.UTF-8
LC_ALL
LANGUAGE

fastlane gems

Gem Version Update-Status
fastlane 2.128.1 ✅ Up-To-Date

Loaded fastlane plugins:

Plugin Version Update-Status
fastlane-plugin-appcenter 1.2.1 ✅ Up-To-Date

Loaded gems

Gem Version
bundler 1.16.6
io-console 0.4.5
rake 12.3.2
CFPropertyList 3.0.0
concurrent-ruby 1.1.5
i18n 0.9.5
minitest 5.11.3
thread_safe 0.3.6
tzinfo 1.2.5
activesupport 4.2.11.1
public_suffix 2.0.5
addressable 2.6.0
eventmachine 1.2.7
websocket-extensions 0.1.4
websocket-driver 0.7.1
faye-websocket 0.10.9
childprocess 1.0.1
rubyzip 1.2.3
selenium-webdriver 3.142.3
appiumlibcore 3.2.0
mini_portile2 2.4.0
nokogiri 1.10.3
tomlrb 1.2.8
appium_lib 10.3.1
atomos 0.1.3
babosa 1.0.2
claide 1.0.3
colored 1.2
colored2 3.1.2
colorize 0.8.1
highline 1.7.10
commander-fastlane 4.4.6
declarative 0.0.10
declarative-option 0.1.0
diff-lcs 1.3
digest-crc 0.4.1
unf_ext 0.0.7.6
unf 0.1.4
domain_name 0.5.20190701
dotenv 2.7.5
emoji_regex 1.0.1
excon 0.66.0
multipart-post 2.0.0
faraday 0.15.4
http-cookie 1.0.3
faraday-cookie_jar 0.0.6
faraday_middleware 0.13.1
fastimage 2.1.5
gh_inspector 1.1.3
jwt 2.1.0
memoist 0.16.0
multi_json 1.13.1
os 1.0.1
signet 0.11.0
googleauth 0.6.7
httpclient 2.8.3
mime-types-data 3.2019.0331
mime-types 3.2.2
uber 0.1.0
representable 3.0.4
retriable 3.1.2
google-api-client 0.23.9
google-cloud-env 1.2.0
google-cloud-core 1.3.0
google-cloud-storage 1.16.0
json 2.2.0
mini_magick 4.9.5
multi_xml 0.6.0
plist 3.5.0
security 0.1.3
naturally 2.2.0
simctl 1.6.5
slack-notifier 2.3.2
terminal-notifier 2.0.0
unicode-display_width 1.6.0
terminal-table 1.8.0
tty-screen 0.7.0
tty-cursor 0.7.0
tty-spinner 0.9.1
word_wrap 1.0.0
nanaimo 0.2.6
xcodeproj 1.12.0
rouge 2.0.7
xcpretty 0.3.0
xcpretty-travis-formatter 1.0.0
fastlane-plugin-appcenter 1.2.1
handlebars-source 4.0.14
libv8 3.16.14.19
ref 2.0.0
therubyracer 0.12.3
handlebars 0.8.0
netrc 0.11.0
rest-client 2.0.2
rspec-support 3.8.2
rspec-core 3.8.2
rspec-expectations 3.8.4
rspec-mocks 3.8.1
rspec 3.8.0
rubyXL 3.4.5
ruby_rncryptor 3.0.2

generated on: 2019-08-07

Asked Oct 19 '21 05:10
avatar sdjmchattie
sdjmchattie

9 Answer:

This file reproduces the error by including sh('cat text.txt') in the Fastfile.

text.txt

1
Answered Aug 08 '19 at 11:55
avatar  of sdjmchattie
sdjmchattie

I can confirm the bug still exists in Fastlane version 2.134.0

1
Answered Oct 18 '19 at 08:56
avatar  of sdjmchattie
sdjmchattie

I can confirm the bug still exists in Fastlane version 2.130.0

1
Answered Sep 09 '19 at 07:58
avatar  of sdjmchattie
sdjmchattie

Sorry yes, the test lane is below:

  lane :test do
    extract_strings
    ensure_git_status_clean
    scan(
      scheme: build_scheme_name,
      slack_channel: slack_channel,
      slack_message: "Branch #{git_branch}",
      slack_only_on_failure: true
    )
  end

where extract_strings is:

  lane :extract_strings do
    sh "cd .. && ./scripts/extract-strings.sh #{project_name}"
    sh "cd .. && ./scripts/strings_tool.rb --patch --project #{project_name}"
  end

And the script crashing fastlane is the extract-strings.sh one which contains the command (truncated to remove unrelated things):

extracted_strings=$1/ExtractedStrings/

echo "> Extracting strings from code"
find "$1" \( -name "*.m" -o -name "*.mm" -o -name "*.swift" \) -print0 | xargs -0 xcrun extractLocStrings -o "$extracted_strings"
1
Answered Aug 08 '19 at 09:04
avatar  of sdjmchattie
sdjmchattie

Ok, so this is a problem with the output the command is giving to sh. Can you share what it returns as a file so we can reproduce with sh('cat file.txt') or similar?

1
Answered Aug 08 '19 at 10:32
avatar  of janpio
janpio

There hasn't been any activity on this issue recently. Due to the high number of incoming GitHub notifications, we have to clean some of the old issues, as many of them have already been resolved with the latest updates.

Please make sure to update to the latest fastlane version and check if that solves the issue. Let us know if that works for you by adding a comment :+1:

Friendly reminder: contributions are always welcome! Check out CONTRIBUTING.md for more information on how to help with fastlane and feel free to tackle this issue yourself :muscle:

1
Answered Sep 07 '19 at 17:31
avatar  of fastlane-bot
fastlane-bot

There hasn't been any activity on this issue recently. Due to the high number of incoming GitHub notifications, we have to clean some of the old issues, as many of them have already been resolved with the latest updates.

Please make sure to update to the latest fastlane version and check if that solves the issue. Let us know if that works for you by adding a comment :+1:

Friendly reminder: contributions are always welcome! Check out CONTRIBUTING.md for more information on how to help with fastlane and feel free to tackle this issue yourself :muscle:

1
Answered Oct 17 '19 at 17:31
avatar  of fastlane-bot
fastlane-bot

There hasn't been any activity on this issue recently. Due to the high number of incoming GitHub notifications, we have to clean some of the old issues, as many of them have already been resolved with the latest updates.

Please make sure to update to the latest fastlane version and check if that solves the issue. Let us know if that works for you by adding a comment :+1:

Friendly reminder: contributions are always welcome! Check out CONTRIBUTING.md for more information on how to help with fastlane and feel free to tackle this issue yourself :muscle:

1
Answered Nov 26 '19 at 17:32
avatar  of fastlane-bot
fastlane-bot

I can confirm the bug still exists in Fastlane version 2.137.0

1
Answered Nov 27 '19 at 09:53
avatar  of sdjmchattie
sdjmchattie