Skip to content

lldb TestMultipleDebuggers.py sometimes fails on Linux #101162

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
DavidSpickett opened this issue Jul 30, 2024 · 4 comments
Open

lldb TestMultipleDebuggers.py sometimes fails on Linux #101162

DavidSpickett opened this issue Jul 30, 2024 · 4 comments
Labels
lldb test-suite

Comments

@DavidSpickett
Copy link
Collaborator

As seen on https://lab.llvm.org/buildbot/#/builders/59/builds/1324 and in Github CI runs. Sometimes this test times out:

TIMEOUT: lldb-api :: api/multiple-debuggers/TestMultipleDebuggers.py (1990 of 1990)
******************** TEST 'lldb-api :: api/multiple-debuggers/TestMultipleDebuggers.py' FAILED ********************
Script:
--
/usr/bin/python3.8 /home/tcwg-buildbot/worker/lldb-aarch64-ubuntu/llvm-project/lldb/test/API/dotest.py -u CXXFLAGS -u CFLAGS --env ARCHIVER=/usr/local/bin/llvm-ar --env OBJCOPY=/usr/bin/llvm-objcopy --env LLVM_LIBS_DIR=/home/tcwg-buildbot/worker/lldb-aarch64-ubuntu/build/./lib --env LLVM_INCLUDE_DIR=/home/tcwg-buildbot/worker/lldb-aarch64-ubuntu/build/include --env LLVM_TOOLS_DIR=/home/tcwg-buildbot/worker/lldb-aarch64-ubuntu/build/./bin --arch aarch64 --build-dir /home/tcwg-buildbot/worker/lldb-aarch64-ubuntu/build/lldb-test-build.noindex --lldb-module-cache-dir /home/tcwg-buildbot/worker/lldb-aarch64-ubuntu/build/lldb-test-build.noindex/module-cache-lldb/lldb-api --clang-module-cache-dir /home/tcwg-buildbot/worker/lldb-aarch64-ubuntu/build/lldb-test-build.noindex/module-cache-clang/lldb-api --executable /home/tcwg-buildbot/worker/lldb-aarch64-ubuntu/build/./bin/lldb --compiler /home/tcwg-buildbot/worker/lldb-aarch64-ubuntu/build/./bin/clang --dsymutil /home/tcwg-buildbot/worker/lldb-aarch64-ubuntu/build/./bin/dsymutil --llvm-tools-dir /home/tcwg-buildbot/worker/lldb-aarch64-ubuntu/build/./bin --lldb-obj-root /home/tcwg-buildbot/worker/lldb-aarch64-ubuntu/build/tools/lldb --lldb-libs-dir /home/tcwg-buildbot/worker/lldb-aarch64-ubuntu/build/./lib /home/tcwg-buildbot/worker/lldb-aarch64-ubuntu/llvm-project/lldb/test/API/api/multiple-debuggers -p TestMultipleDebuggers.py
--
Exit Code: -9
Timeout: Reached timeout of 600 seconds
Command Output (stdout):
--
lldb version 19.0.0git (https://github.com/llvm/llvm-project.git revision a4a8d36b8591801696aad30fa116669c7bb2348c)
  clang revision a4a8d36b8591801696aad30fa116669c7bb2348c
  llvm revision a4a8d36b8591801696aad30fa116669c7bb2348c
--
********************
********************
Timed Out Tests (1):
  lldb-api :: api/multiple-debuggers/TestMultipleDebuggers.py

My impression is that some change the 9th of July timeframe made this more common, but equally, it could just be a shift in the load on the bots themselves.

Also, running this on AArch64 I realised that none of the child processes start properly so it needs restricting to x86_64 only, or to use the host architecture name when starting the debugger.
DavidSpickett added a commit to DavidSpickett/llvm-project that referenced this issue Jul 30, 2024
@DavidSpickett
[lldb][test] Fix TestMultipleDebuggers test on non-x86, other small i…
491bf0e 
…ssues

This test has been flaky lately (llvm#101162)
and I disabled it everywhere initially.

I found that it always uses "x86_64" for the program architecture so the test
was "passing" elsewhere but I don't think it was meant to. So I have added a define
to pass on the host's architecture when compiling. This makes it work on AArch64
as well.

While I'm here I've fixed the uint64_t formatting warnings by using the defined
formats that'll work everywhere.

In addition, I found that the function names include "()" on Linux, so now we
check for "foo" or "foo()".

The test cpp file has never been formatted so I've not done that either, just
kept to the local style.

I've removed the Linux skip to see if any of this helps the timeouts, and to
verify the build command changes. If the timeouts come back I'll disable it again.
@DavidSpickett DavidSpickett mentioned this issue Jul 30, 2024
Merged
DavidSpickett referenced this issue Jul 30, 2024
@DavidSpickett
[lldb][test] Disable flakey TestMultipleDebuggers on Linux
8210087 
This has been flakey lately on Linux on the buildbots and in CI
on PRs.

For example:
https://lab.llvm.org/buildbot/#/builders/59/builds/1324
#98237

I will try to track down the cause tomorrow.
@DavidSpickett
Copy link
Collaborator Author

#101169 to update the test to work properly on AArch64 and re-enable it.

I wasn't able to reproduce the timeout myself so if it happens again I'll disable it again. It is under a "stresstest" category, so losing Linux coverage here isn't the end of the world.

@llvmbot
Copy link
Member

llvmbot commented Jul 30, 2024

@llvm/issue-subscribers-lldb

Author: David Spickett (DavidSpickett)

As seen on https://lab.llvm.org/buildbot/#/builders/59/builds/1324 and in Github CI runs. Sometimes this test times out: ``` TIMEOUT: lldb-api :: api/multiple-debuggers/TestMultipleDebuggers.py (1990 of 1990) ******************** TEST 'lldb-api :: api/multiple-debuggers/TestMultipleDebuggers.py' FAILED ******************** Script: -- /usr/bin/python3.8 /home/tcwg-buildbot/worker/lldb-aarch64-ubuntu/llvm-project/lldb/test/API/dotest.py -u CXXFLAGS -u CFLAGS --env ARCHIVER=/usr/local/bin/llvm-ar --env OBJCOPY=/usr/bin/llvm-objcopy --env LLVM_LIBS_DIR=/home/tcwg-buildbot/worker/lldb-aarch64-ubuntu/build/./lib --env LLVM_INCLUDE_DIR=/home/tcwg-buildbot/worker/lldb-aarch64-ubuntu/build/include --env LLVM_TOOLS_DIR=/home/tcwg-buildbot/worker/lldb-aarch64-ubuntu/build/./bin --arch aarch64 --build-dir /home/tcwg-buildbot/worker/lldb-aarch64-ubuntu/build/lldb-test-build.noindex --lldb-module-cache-dir /home/tcwg-buildbot/worker/lldb-aarch64-ubuntu/build/lldb-test-build.noindex/module-cache-lldb/lldb-api --clang-module-cache-dir /home/tcwg-buildbot/worker/lldb-aarch64-ubuntu/build/lldb-test-build.noindex/module-cache-clang/lldb-api --executable /home/tcwg-buildbot/worker/lldb-aarch64-ubuntu/build/./bin/lldb --compiler /home/tcwg-buildbot/worker/lldb-aarch64-ubuntu/build/./bin/clang --dsymutil /home/tcwg-buildbot/worker/lldb-aarch64-ubuntu/build/./bin/dsymutil --llvm-tools-dir /home/tcwg-buildbot/worker/lldb-aarch64-ubuntu/build/./bin --lldb-obj-root /home/tcwg-buildbot/worker/lldb-aarch64-ubuntu/build/tools/lldb --lldb-libs-dir /home/tcwg-buildbot/worker/lldb-aarch64-ubuntu/build/./lib /home/tcwg-buildbot/worker/lldb-aarch64-ubuntu/llvm-project/lldb/test/API/api/multiple-debuggers -p TestMultipleDebuggers.py -- Exit Code: -9 Timeout: Reached timeout of 600 seconds Command Output (stdout): -- lldb version 19.0.0git (https://github.com/llvm/llvm-project.git revision a4a8d36) clang revision a4a8d36 llvm revision a4a8d36 -- ******************** ******************** Timed Out Tests (1): lldb-api :: api/multiple-debuggers/TestMultipleDebuggers.py ```

My impression is that some change the 9th of July timeframe made this more common, but equally, it could just be a shift in the load on the bots themselves.

Also, running this on AArch64 I realised that none of the child processes start properly so it needs restricting to x86_64 only, or to use the host architecture name when starting the debugger.

DavidSpickett added a commit that referenced this issue Jul 31, 2024
@DavidSpickett
[lldb][test] Fix TestMultipleDebuggers test on non-x86, other small i…
aa07282 
…ssues (#101169)

This test has been flaky lately
(#101162) and I disabled it
everywhere initially.

I found that it always uses "x86_64" for the program architecture so the
test was "passing" elsewhere but I don't think it was meant to. So I
have added a define to pass on the host's architecture when compiling.
This makes it work on AArch64 as well.

While I'm here I've fixed the uint64_t formatting warnings by using the
defined formats that'll work everywhere.

In addition, I found that the function names include "()" on Linux, so
now we check for "foo" or "foo()".

The test cpp file has never been, or was only partially formatted so
I've not formatted the changes, just kept to the local style.

I've removed the Linux skip to see if any of this helps the timeouts,
and to verify the build command changes. If the timeouts come back I'll
disable it again.
@DavidSpickett
Copy link
Collaborator Author

1 report in the last week from AArch64 Linux, but it failed to set up the test instead of timing out:

FAIL: LLDB (/home/tcwg-buildbot/worker/lldb-aarch64-ubuntu/build/bin/clang-aarch64) :: test_multiple_debuggers (TestMultipleDebuggers.TestMultipleSimultaneousDebuggers)
======================================================================
ERROR: test_multiple_debuggers (TestMultipleDebuggers.TestMultipleSimultaneousDebuggers)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/tcwg-buildbot/worker/lldb-aarch64-ubuntu/llvm-project/lldb/packages/Python/lldbsuite/test/decorators.py", line 148, in wrapper
    return func(*args, **kwargs)
  File "/home/tcwg-buildbot/worker/lldb-aarch64-ubuntu/llvm-project/lldb/packages/Python/lldbsuite/test/decorators.py", line 148, in wrapper
    return func(*args, **kwargs)
  File "/home/tcwg-buildbot/worker/lldb-aarch64-ubuntu/llvm-project/lldb/test/API/api/multiple-debuggers/TestMultipleDebuggers.py", line 35, in test_multiple_debuggers
    subprocess.check_call([self.driver_exe, self.inferior_exe])
  File "/usr/lib/python3.8/subprocess.py", line 364, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['/home/tcwg-buildbot/worker/lldb-aarch64-ubuntu/build/lldb-test-build.noindex/api/multiple-debuggers/TestMultipleDebuggers.test_multiple_debuggers/multi-process-driver', '/home/tcwg-buildbot/worker/lldb-aarch64-ubuntu/build/lldb-test-build.noindex/api/multiple-debuggers/TestMultipleDebuggers.test_multiple_debuggers/testprog']' died with <Signals.SIGSEGV: 11>.
Config=aarch64-/home/tcwg-buildbot/worker/lldb-aarch64-ubuntu/build/bin/clang
----------------------------------------------------------------------

https://lab.llvm.org/buildbot/#/builders/59/builds/2798

I've never seen this before, it only happened once, and it's not the timeout issue that was much more common previously. So I will close this issue, the link to it remains in the test file for reference if these timeouts return.

@DavidSpickett DavidSpickett mentioned this issue Sep 4, 2024
Closed
@DavidSpickett DavidSpickett reopened this Sep 4, 2024
@DavidSpickett DavidSpickett changed the title lldb TestMultipleDebuggers.py sometimes times out on Linux lldb TestMultipleDebuggers.py sometimes fails on Linux Sep 4, 2024
@DavidSpickett
Copy link
Collaborator Author

It's back again but with SIGABRT instead:
#105765 (comment)

DavidSpickett added a commit that referenced this issue Sep 4, 2024
@DavidSpickett
[lldb][test] Disable TestMultipleDebuggers on Linux
0b2550f 
This used to timeout (#101162)
now it's aborting (#105765 (comment))
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
lldb test-suite
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@DavidSpickett @llvmbot