Orbiter History

checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..........................................X...X.............] duration: 6.210459803s trace: https://jaeger.dev.storj.io/trace/74cf1e30f6d49950 FAIL (2025-11-05 19:02:52 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.21050086s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [........................................X..X................] duration: 5.722221679s trace: https://jaeger.dev.storj.io/trace/3e2ae8884260b157 FAIL (2025-11-05 19:02:46 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.72225831s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......................................X...X................] duration: 5.879058962s trace: https://jaeger.dev.storj.io/trace/2e576aae48e8f03e FAIL (2025-11-05 19:02:37 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== 5 18:52:06.850167887 (151h11m38.54601663s) [noise=true , address=145.79.227.25:4618 , piece_id=XRUPY5BUPG7BXN2MI6MGYO4N3GW6TZDJE46MZFNRELWANHHG3N5A, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.850747301 (151h11m38.546596054s) [noise=true , address=77.90.31.15:4698 , piece_id=7VACL26N7ZCBXIA2I3XL5Q3UJ2BAHDIPXX4VZCQX3S63ABNW625Q, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.851222605 (151h11m38.547071358s) [noise=true , address=145.79.227.17:4972 , piece_id=H7UHQV7RDLCZUHY2XMXUAD2PX4ZBLWKTM47N6PH7UOGBRXPMWVOA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.851558575 (151h11m38.547407319s) [noise=true , address=145.79.227.22:4509 , piece_id=7RA5NJ72PI3DT4IC2ECVLZ3A2HXXL546ELVDXHZW4OXNN4D36OEA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.851909263 (151h11m38.547758006s) [noise=true , address=145.79.227.26:4654 , piece_id=E5J22KNPQSETSY443QQZXFFKUUJFHWFV22SQAVZ4BFQ4M6YWCNQQ, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.852327540 (151h11m38.548176293s) [noise=true , address=45.91.116.7:4010 , piece_id=EBVGGNQF3N4MTUIATJUWK4QZN7SEOWI6SIAW6N7JZOB5G6H33NEQ, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.852728624 (151h11m38.548577377s) [noise=true , address=145.79.227.9:4366 , piece_id=7ZRECKBQC4S6QLERBZY442WYS4H35XWOZYSGGPHSU7NQ6WL4UNVA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.853978774 (151h11m38.549827528s) [noise=true , address=77.90.31.18:4805 , piece_id=U2U34KK4AJK4A5HGNTHEYCIARLAR4X5NLOVCF63TN3PJUIRMMIHA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.854447296 (151h11m38.550296059s) [noise=true , address=77.90.31.16:4734 , piece_id=HFWVFM46PZFLZT6YNZVIA6HKLVN2PTV2GETZEKX356DMDGQO4HOA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.856523089 (151h11m38.552371842s) [noise=true , address=45.91.116.11:4117 , piece_id=QUGUYHEJWEEV733CBC3ONMV7XWDKVH2OFLQWXPQHPKZ7GTH4RLUA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.859490379 (151h11m38.555339122s) [noise=true , address=77.90.31.13:4628 , piece_id=QFTEHZ5FY7A5DPZZ44TRXGFRHPRNJZJHLQZDGRGOWYFEZYIZT2NQ, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.861350291 (151h11m38.557199034s) [noise=true , address=145.79.227.31:4797 , piece_id=2TFSZNURIAU573QUA5DUS7AJZNY2OXC6TJL3ST4VXIWY2WVB5YJA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.863368665 (151h11m38.559217418s) [noise=true , address=145.79.227.23:4547 , piece_id=K54H437KW5AAYUWOTMDBDP5GJ3ZABFKLAAOSS6WS5G5CROYEHODQ, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.864848794 (151h11m38.560697547s) [noise=true , address=145.79.227.24:4581 , piece_id=TSCVCH2KJEVMD4AVOEIHGTMRCRZ6NZMZA2CTQ6FUO2EIL3JAVYEA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.865644930 (151h11m38.561493683s) [noise=true , address=145.79.227.8:4330 , piece_id=4A47HIUX2MTJ2673XZZCMMF4T235TS4JWMNYJEVHHCKAHQX6APMA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:08.560504137 (151h11m40.25635289s) [upload=133242]: Next returned nil segment Nov 5 18:52:08.560522693 (151h11m40.256371446s) [upload=133242]: Waiting for error group managing segments... Nov 5 18:52:08.850333179 (151h11m40.546181932s) [noise=true , address=145.79.227.17:4972 , piece_id=H7UHQV7RDLCZUHY2XMXUAD2PX4ZBLWKTM47N6PH7UOGBRXPMWVOA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.853517822 (151h11m40.549366575s) [noise=true , address=145.79.227.22:4509 , piece_id=7RA5NJ72PI3DT4IC2ECVLZ3A2HXXL546ELVDXHZW4OXNN4D36OEA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.855547508 (151h11m40.551396261s) [noise=true , address=145.79.227.9:4366 , piece_id=7ZRECKBQC4S6QLERBZY442WYS4H35XWOZYSGGPHSU7NQ6WL4UNVA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.856103106 (151h11m40.551951859s) [noise=true , address=145.79.227.25:4618 , piece_id=XRUPY5BUPG7BXN2MI6MGYO4N3GW6TZDJE46MZFNRELWANHHG3N5A, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.860417321 (151h11m40.556266064s) [noise=true , address=145.79.227.8:4330 , piece_id=4A47HIUX2MTJ2673XZZCMMF4T235TS4JWMNYJEVHHCKAHQX6APMA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.865602796 (151h11m40.561451549s) [noise=true , address=145.79.227.23:4547 , piece_id=K54H437KW5AAYUWOTMDBDP5GJ3ZABFKLAAOSS6WS5G5CROYEHODQ, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.879393757 (151h11m40.57524251s) [noise=true , address=145.79.227.31:4797 , piece_id=2TFSZNURIAU573QUA5DUS7AJZNY2OXC6TJL3ST4VXIWY2WVB5YJA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.905541449 (151h11m40.601390202s) [noise=true , address=145.79.227.26:4654 , piece_id=E5J22KNPQSETSY443QQZXFFKUUJFHWFV22SQAVZ4BFQ4M6YWCNQQ, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.929018836 (151h11m40.624867589s) [noise=true , address=145.79.227.24:4581 , piece_id=TSCVCH2KJEVMD4AVOEIHGTMRCRZ6NZMZA2CTQ6FUO2EIL3JAVYEA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:35.899554917 (151h12m7.59540368s) [noise=true , address=77.90.31.17:4771 , piece_id=KVFE7VYW3RZJ25BWSH4BIRRQCMWDYOIK7EJ62XVREGN26YN7XYMQ, seg_pos={0 0}, upload=133242]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: write tcp 188.166.225.170:60676->77.90.31.17:4771: use of closed network connection; write tcp 188.166.225.170:60676->77.90.31.17:4771: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60676->77.90.31.17:4771: use of closed network connection Nov 5 18:52:35.899727506 (151h12m7.595576259s) [noise=true , address=77.90.31.13:4628 , piece_id=QFTEHZ5FY7A5DPZZ44TRXGFRHPRNJZJHLQZDGRGOWYFEZYIZT2NQ, seg_pos={0 0}, upload=133242]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: write tcp 188.166.225.170:42026->77.90.31.13:4628: use of closed network connection; write tcp 188.166.225.170:42026->77.90.31.13:4628: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42026->77.90.31.13:4628: use of closed network connection Nov 5 18:52:35.899860439 (151h12m7.595709192s) [noise=true , address=45.91.116.7:4010 , piece_id=EBVGGNQF3N4MTUIATJUWK4QZN7SEOWI6SIAW6N7JZOB5G6H33NEQ, seg_pos={0 0}, upload=133242]: Done uploading piece. err: ecclient: upload failed (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, address:45.91.116.7:4010): protocol: write tcp 188.166.225.170:40036->45.91.116.7:4010: use of closed network connection; write tcp 188.166.225.170:40036->45.91.116.7:4010: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40036->45.91.116.7:4010: use of closed network connection Nov 5 18:52:35.899923829 (151h12m7.595772572s) [noise=true , address=77.90.31.18:4805 , piece_id=U2U34KK4AJK4A5HGNTHEYCIARLAR4X5NLOVCF63TN3PJUIRMMIHA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: write tcp 188.166.225.170:60758->77.90.31.18:4805: use of closed network connection; write tcp 188.166.225.170:60758->77.90.31.18:4805: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60758->77.90.31.18:4805: use of closed network connection Nov 5 18:52:35.900052084 (151h12m7.595900837s) [noise=true , address=77.90.31.16:4734 , piece_id=HFWVFM46PZFLZT6YNZVIA6HKLVN2PTV2GETZEKX356DMDGQO4HOA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: write tcp 188.166.225.170:50728->77.90.31.16:4734: use of closed network connection; write tcp 188.166.225.170:50728->77.90.31.16:4734: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50728->77.90.31.16:4734: use of closed network connection Nov 5 18:52:35.900231175 (151h12m7.596079938s) [noise=true , address=45.91.116.11:4117 , piece_id=QUGUYHEJWEEV733CBC3ONMV7XWDKVH2OFLQWXPQHPKZ7GTH4RLUA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: ecclient: upload failed (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, address:45.91.116.11:4117): protocol: write tcp 188.166.225.170:41292->45.91.116.11:4117: use of closed network connection; write tcp 188.166.225.170:41292->45.91.116.11:4117: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41292->45.91.116.11:4117: use of closed network connection Nov 5 18:52:35.900342116 (151h12m7.596190859s) [noise=true , address=77.90.31.15:4698 , piece_id=7VACL26N7ZCBXIA2I3XL5Q3UJ2BAHDIPXX4VZCQX3S63ABNW625Q, seg_pos={0 0}, upload=133242]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: write tcp 188.166.225.170:41118->77.90.31.15:4698: use of closed network connection; write tcp 188.166.225.170:41118->77.90.31.15:4698: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41118->77.90.31.15:4698: use of closed network connection Nov 5 18:52:35.900382814 (151h12m7.596231567s) [seg_pos={0 0}, upload=133242]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Nov 5 18:52:36.162442033 (151h12m7.858290786s) [upload=133242]: Done uploading segments... Nov 5 18:52:36.162481568 (151h12m7.858330321s) [upload=133242]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 5.879092586s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [.........................................XX...X.............] duration: 3.477929525s trace: https://jaeger.dev.storj.io/trace/204ab2570dc82d9c FAIL (2025-11-05 19:02:37 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.477949262s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...........................................X...X............] duration: 6.258616616s trace: https://jaeger.dev.storj.io/trace/74cf1e30f6d49950 FAIL (2025-11-05 19:02:27 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.258656052s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.........................................X..X...............] duration: 5.843615832s trace: https://jaeger.dev.storj.io/trace/3e2ae8884260b157 FAIL (2025-11-05 19:02:20 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.843649388s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [........................................X...X...............] duration: 6.016610761s trace: https://jaeger.dev.storj.io/trace/2e576aae48e8f03e FAIL (2025-11-05 19:01:55 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== 5 18:52:06.850167887 (151h11m38.54601663s) [noise=true , address=145.79.227.25:4618 , piece_id=XRUPY5BUPG7BXN2MI6MGYO4N3GW6TZDJE46MZFNRELWANHHG3N5A, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.850747301 (151h11m38.546596054s) [noise=true , address=77.90.31.15:4698 , piece_id=7VACL26N7ZCBXIA2I3XL5Q3UJ2BAHDIPXX4VZCQX3S63ABNW625Q, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.851222605 (151h11m38.547071358s) [noise=true , address=145.79.227.17:4972 , piece_id=H7UHQV7RDLCZUHY2XMXUAD2PX4ZBLWKTM47N6PH7UOGBRXPMWVOA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.851558575 (151h11m38.547407319s) [noise=true , address=145.79.227.22:4509 , piece_id=7RA5NJ72PI3DT4IC2ECVLZ3A2HXXL546ELVDXHZW4OXNN4D36OEA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.851909263 (151h11m38.547758006s) [noise=true , address=145.79.227.26:4654 , piece_id=E5J22KNPQSETSY443QQZXFFKUUJFHWFV22SQAVZ4BFQ4M6YWCNQQ, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.852327540 (151h11m38.548176293s) [noise=true , address=45.91.116.7:4010 , piece_id=EBVGGNQF3N4MTUIATJUWK4QZN7SEOWI6SIAW6N7JZOB5G6H33NEQ, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.852728624 (151h11m38.548577377s) [noise=true , address=145.79.227.9:4366 , piece_id=7ZRECKBQC4S6QLERBZY442WYS4H35XWOZYSGGPHSU7NQ6WL4UNVA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.853978774 (151h11m38.549827528s) [noise=true , address=77.90.31.18:4805 , piece_id=U2U34KK4AJK4A5HGNTHEYCIARLAR4X5NLOVCF63TN3PJUIRMMIHA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.854447296 (151h11m38.550296059s) [noise=true , address=77.90.31.16:4734 , piece_id=HFWVFM46PZFLZT6YNZVIA6HKLVN2PTV2GETZEKX356DMDGQO4HOA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.856523089 (151h11m38.552371842s) [noise=true , address=45.91.116.11:4117 , piece_id=QUGUYHEJWEEV733CBC3ONMV7XWDKVH2OFLQWXPQHPKZ7GTH4RLUA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.859490379 (151h11m38.555339122s) [noise=true , address=77.90.31.13:4628 , piece_id=QFTEHZ5FY7A5DPZZ44TRXGFRHPRNJZJHLQZDGRGOWYFEZYIZT2NQ, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.861350291 (151h11m38.557199034s) [noise=true , address=145.79.227.31:4797 , piece_id=2TFSZNURIAU573QUA5DUS7AJZNY2OXC6TJL3ST4VXIWY2WVB5YJA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.863368665 (151h11m38.559217418s) [noise=true , address=145.79.227.23:4547 , piece_id=K54H437KW5AAYUWOTMDBDP5GJ3ZABFKLAAOSS6WS5G5CROYEHODQ, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.864848794 (151h11m38.560697547s) [noise=true , address=145.79.227.24:4581 , piece_id=TSCVCH2KJEVMD4AVOEIHGTMRCRZ6NZMZA2CTQ6FUO2EIL3JAVYEA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.865644930 (151h11m38.561493683s) [noise=true , address=145.79.227.8:4330 , piece_id=4A47HIUX2MTJ2673XZZCMMF4T235TS4JWMNYJEVHHCKAHQX6APMA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:08.560504137 (151h11m40.25635289s) [upload=133242]: Next returned nil segment Nov 5 18:52:08.560522693 (151h11m40.256371446s) [upload=133242]: Waiting for error group managing segments... Nov 5 18:52:08.850333179 (151h11m40.546181932s) [noise=true , address=145.79.227.17:4972 , piece_id=H7UHQV7RDLCZUHY2XMXUAD2PX4ZBLWKTM47N6PH7UOGBRXPMWVOA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.853517822 (151h11m40.549366575s) [noise=true , address=145.79.227.22:4509 , piece_id=7RA5NJ72PI3DT4IC2ECVLZ3A2HXXL546ELVDXHZW4OXNN4D36OEA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.855547508 (151h11m40.551396261s) [noise=true , address=145.79.227.9:4366 , piece_id=7ZRECKBQC4S6QLERBZY442WYS4H35XWOZYSGGPHSU7NQ6WL4UNVA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.856103106 (151h11m40.551951859s) [noise=true , address=145.79.227.25:4618 , piece_id=XRUPY5BUPG7BXN2MI6MGYO4N3GW6TZDJE46MZFNRELWANHHG3N5A, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.860417321 (151h11m40.556266064s) [noise=true , address=145.79.227.8:4330 , piece_id=4A47HIUX2MTJ2673XZZCMMF4T235TS4JWMNYJEVHHCKAHQX6APMA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.865602796 (151h11m40.561451549s) [noise=true , address=145.79.227.23:4547 , piece_id=K54H437KW5AAYUWOTMDBDP5GJ3ZABFKLAAOSS6WS5G5CROYEHODQ, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.879393757 (151h11m40.57524251s) [noise=true , address=145.79.227.31:4797 , piece_id=2TFSZNURIAU573QUA5DUS7AJZNY2OXC6TJL3ST4VXIWY2WVB5YJA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.905541449 (151h11m40.601390202s) [noise=true , address=145.79.227.26:4654 , piece_id=E5J22KNPQSETSY443QQZXFFKUUJFHWFV22SQAVZ4BFQ4M6YWCNQQ, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.929018836 (151h11m40.624867589s) [noise=true , address=145.79.227.24:4581 , piece_id=TSCVCH2KJEVMD4AVOEIHGTMRCRZ6NZMZA2CTQ6FUO2EIL3JAVYEA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:35.899554917 (151h12m7.59540368s) [noise=true , address=77.90.31.17:4771 , piece_id=KVFE7VYW3RZJ25BWSH4BIRRQCMWDYOIK7EJ62XVREGN26YN7XYMQ, seg_pos={0 0}, upload=133242]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: write tcp 188.166.225.170:60676->77.90.31.17:4771: use of closed network connection; write tcp 188.166.225.170:60676->77.90.31.17:4771: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60676->77.90.31.17:4771: use of closed network connection Nov 5 18:52:35.899727506 (151h12m7.595576259s) [noise=true , address=77.90.31.13:4628 , piece_id=QFTEHZ5FY7A5DPZZ44TRXGFRHPRNJZJHLQZDGRGOWYFEZYIZT2NQ, seg_pos={0 0}, upload=133242]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: write tcp 188.166.225.170:42026->77.90.31.13:4628: use of closed network connection; write tcp 188.166.225.170:42026->77.90.31.13:4628: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42026->77.90.31.13:4628: use of closed network connection Nov 5 18:52:35.899860439 (151h12m7.595709192s) [noise=true , address=45.91.116.7:4010 , piece_id=EBVGGNQF3N4MTUIATJUWK4QZN7SEOWI6SIAW6N7JZOB5G6H33NEQ, seg_pos={0 0}, upload=133242]: Done uploading piece. err: ecclient: upload failed (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, address:45.91.116.7:4010): protocol: write tcp 188.166.225.170:40036->45.91.116.7:4010: use of closed network connection; write tcp 188.166.225.170:40036->45.91.116.7:4010: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40036->45.91.116.7:4010: use of closed network connection Nov 5 18:52:35.899923829 (151h12m7.595772572s) [noise=true , address=77.90.31.18:4805 , piece_id=U2U34KK4AJK4A5HGNTHEYCIARLAR4X5NLOVCF63TN3PJUIRMMIHA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: write tcp 188.166.225.170:60758->77.90.31.18:4805: use of closed network connection; write tcp 188.166.225.170:60758->77.90.31.18:4805: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60758->77.90.31.18:4805: use of closed network connection Nov 5 18:52:35.900052084 (151h12m7.595900837s) [noise=true , address=77.90.31.16:4734 , piece_id=HFWVFM46PZFLZT6YNZVIA6HKLVN2PTV2GETZEKX356DMDGQO4HOA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: write tcp 188.166.225.170:50728->77.90.31.16:4734: use of closed network connection; write tcp 188.166.225.170:50728->77.90.31.16:4734: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50728->77.90.31.16:4734: use of closed network connection Nov 5 18:52:35.900231175 (151h12m7.596079938s) [noise=true , address=45.91.116.11:4117 , piece_id=QUGUYHEJWEEV733CBC3ONMV7XWDKVH2OFLQWXPQHPKZ7GTH4RLUA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: ecclient: upload failed (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, address:45.91.116.11:4117): protocol: write tcp 188.166.225.170:41292->45.91.116.11:4117: use of closed network connection; write tcp 188.166.225.170:41292->45.91.116.11:4117: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41292->45.91.116.11:4117: use of closed network connection Nov 5 18:52:35.900342116 (151h12m7.596190859s) [noise=true , address=77.90.31.15:4698 , piece_id=7VACL26N7ZCBXIA2I3XL5Q3UJ2BAHDIPXX4VZCQX3S63ABNW625Q, seg_pos={0 0}, upload=133242]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: write tcp 188.166.225.170:41118->77.90.31.15:4698: use of closed network connection; write tcp 188.166.225.170:41118->77.90.31.15:4698: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41118->77.90.31.15:4698: use of closed network connection Nov 5 18:52:35.900382814 (151h12m7.596231567s) [seg_pos={0 0}, upload=133242]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Nov 5 18:52:36.162442033 (151h12m7.858290786s) [upload=133242]: Done uploading segments... Nov 5 18:52:36.162481568 (151h12m7.858330321s) [upload=133242]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 6.016635658s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.........................................X...X..............] duration: 6.038329537s trace: https://jaeger.dev.storj.io/trace/2e576aae48e8f03e FAIL (2025-11-05 19:01:37 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== 5 18:52:06.850167887 (151h11m38.54601663s) [noise=true , address=145.79.227.25:4618 , piece_id=XRUPY5BUPG7BXN2MI6MGYO4N3GW6TZDJE46MZFNRELWANHHG3N5A, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.850747301 (151h11m38.546596054s) [noise=true , address=77.90.31.15:4698 , piece_id=7VACL26N7ZCBXIA2I3XL5Q3UJ2BAHDIPXX4VZCQX3S63ABNW625Q, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.851222605 (151h11m38.547071358s) [noise=true , address=145.79.227.17:4972 , piece_id=H7UHQV7RDLCZUHY2XMXUAD2PX4ZBLWKTM47N6PH7UOGBRXPMWVOA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.851558575 (151h11m38.547407319s) [noise=true , address=145.79.227.22:4509 , piece_id=7RA5NJ72PI3DT4IC2ECVLZ3A2HXXL546ELVDXHZW4OXNN4D36OEA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.851909263 (151h11m38.547758006s) [noise=true , address=145.79.227.26:4654 , piece_id=E5J22KNPQSETSY443QQZXFFKUUJFHWFV22SQAVZ4BFQ4M6YWCNQQ, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.852327540 (151h11m38.548176293s) [noise=true , address=45.91.116.7:4010 , piece_id=EBVGGNQF3N4MTUIATJUWK4QZN7SEOWI6SIAW6N7JZOB5G6H33NEQ, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.852728624 (151h11m38.548577377s) [noise=true , address=145.79.227.9:4366 , piece_id=7ZRECKBQC4S6QLERBZY442WYS4H35XWOZYSGGPHSU7NQ6WL4UNVA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.853978774 (151h11m38.549827528s) [noise=true , address=77.90.31.18:4805 , piece_id=U2U34KK4AJK4A5HGNTHEYCIARLAR4X5NLOVCF63TN3PJUIRMMIHA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.854447296 (151h11m38.550296059s) [noise=true , address=77.90.31.16:4734 , piece_id=HFWVFM46PZFLZT6YNZVIA6HKLVN2PTV2GETZEKX356DMDGQO4HOA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.856523089 (151h11m38.552371842s) [noise=true , address=45.91.116.11:4117 , piece_id=QUGUYHEJWEEV733CBC3ONMV7XWDKVH2OFLQWXPQHPKZ7GTH4RLUA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.859490379 (151h11m38.555339122s) [noise=true , address=77.90.31.13:4628 , piece_id=QFTEHZ5FY7A5DPZZ44TRXGFRHPRNJZJHLQZDGRGOWYFEZYIZT2NQ, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.861350291 (151h11m38.557199034s) [noise=true , address=145.79.227.31:4797 , piece_id=2TFSZNURIAU573QUA5DUS7AJZNY2OXC6TJL3ST4VXIWY2WVB5YJA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.863368665 (151h11m38.559217418s) [noise=true , address=145.79.227.23:4547 , piece_id=K54H437KW5AAYUWOTMDBDP5GJ3ZABFKLAAOSS6WS5G5CROYEHODQ, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.864848794 (151h11m38.560697547s) [noise=true , address=145.79.227.24:4581 , piece_id=TSCVCH2KJEVMD4AVOEIHGTMRCRZ6NZMZA2CTQ6FUO2EIL3JAVYEA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.865644930 (151h11m38.561493683s) [noise=true , address=145.79.227.8:4330 , piece_id=4A47HIUX2MTJ2673XZZCMMF4T235TS4JWMNYJEVHHCKAHQX6APMA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:08.560504137 (151h11m40.25635289s) [upload=133242]: Next returned nil segment Nov 5 18:52:08.560522693 (151h11m40.256371446s) [upload=133242]: Waiting for error group managing segments... Nov 5 18:52:08.850333179 (151h11m40.546181932s) [noise=true , address=145.79.227.17:4972 , piece_id=H7UHQV7RDLCZUHY2XMXUAD2PX4ZBLWKTM47N6PH7UOGBRXPMWVOA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.853517822 (151h11m40.549366575s) [noise=true , address=145.79.227.22:4509 , piece_id=7RA5NJ72PI3DT4IC2ECVLZ3A2HXXL546ELVDXHZW4OXNN4D36OEA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.855547508 (151h11m40.551396261s) [noise=true , address=145.79.227.9:4366 , piece_id=7ZRECKBQC4S6QLERBZY442WYS4H35XWOZYSGGPHSU7NQ6WL4UNVA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.856103106 (151h11m40.551951859s) [noise=true , address=145.79.227.25:4618 , piece_id=XRUPY5BUPG7BXN2MI6MGYO4N3GW6TZDJE46MZFNRELWANHHG3N5A, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.860417321 (151h11m40.556266064s) [noise=true , address=145.79.227.8:4330 , piece_id=4A47HIUX2MTJ2673XZZCMMF4T235TS4JWMNYJEVHHCKAHQX6APMA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.865602796 (151h11m40.561451549s) [noise=true , address=145.79.227.23:4547 , piece_id=K54H437KW5AAYUWOTMDBDP5GJ3ZABFKLAAOSS6WS5G5CROYEHODQ, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.879393757 (151h11m40.57524251s) [noise=true , address=145.79.227.31:4797 , piece_id=2TFSZNURIAU573QUA5DUS7AJZNY2OXC6TJL3ST4VXIWY2WVB5YJA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.905541449 (151h11m40.601390202s) [noise=true , address=145.79.227.26:4654 , piece_id=E5J22KNPQSETSY443QQZXFFKUUJFHWFV22SQAVZ4BFQ4M6YWCNQQ, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.929018836 (151h11m40.624867589s) [noise=true , address=145.79.227.24:4581 , piece_id=TSCVCH2KJEVMD4AVOEIHGTMRCRZ6NZMZA2CTQ6FUO2EIL3JAVYEA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:35.899554917 (151h12m7.59540368s) [noise=true , address=77.90.31.17:4771 , piece_id=KVFE7VYW3RZJ25BWSH4BIRRQCMWDYOIK7EJ62XVREGN26YN7XYMQ, seg_pos={0 0}, upload=133242]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: write tcp 188.166.225.170:60676->77.90.31.17:4771: use of closed network connection; write tcp 188.166.225.170:60676->77.90.31.17:4771: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60676->77.90.31.17:4771: use of closed network connection Nov 5 18:52:35.899727506 (151h12m7.595576259s) [noise=true , address=77.90.31.13:4628 , piece_id=QFTEHZ5FY7A5DPZZ44TRXGFRHPRNJZJHLQZDGRGOWYFEZYIZT2NQ, seg_pos={0 0}, upload=133242]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: write tcp 188.166.225.170:42026->77.90.31.13:4628: use of closed network connection; write tcp 188.166.225.170:42026->77.90.31.13:4628: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42026->77.90.31.13:4628: use of closed network connection Nov 5 18:52:35.899860439 (151h12m7.595709192s) [noise=true , address=45.91.116.7:4010 , piece_id=EBVGGNQF3N4MTUIATJUWK4QZN7SEOWI6SIAW6N7JZOB5G6H33NEQ, seg_pos={0 0}, upload=133242]: Done uploading piece. err: ecclient: upload failed (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, address:45.91.116.7:4010): protocol: write tcp 188.166.225.170:40036->45.91.116.7:4010: use of closed network connection; write tcp 188.166.225.170:40036->45.91.116.7:4010: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40036->45.91.116.7:4010: use of closed network connection Nov 5 18:52:35.899923829 (151h12m7.595772572s) [noise=true , address=77.90.31.18:4805 , piece_id=U2U34KK4AJK4A5HGNTHEYCIARLAR4X5NLOVCF63TN3PJUIRMMIHA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: write tcp 188.166.225.170:60758->77.90.31.18:4805: use of closed network connection; write tcp 188.166.225.170:60758->77.90.31.18:4805: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60758->77.90.31.18:4805: use of closed network connection Nov 5 18:52:35.900052084 (151h12m7.595900837s) [noise=true , address=77.90.31.16:4734 , piece_id=HFWVFM46PZFLZT6YNZVIA6HKLVN2PTV2GETZEKX356DMDGQO4HOA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: write tcp 188.166.225.170:50728->77.90.31.16:4734: use of closed network connection; write tcp 188.166.225.170:50728->77.90.31.16:4734: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50728->77.90.31.16:4734: use of closed network connection Nov 5 18:52:35.900231175 (151h12m7.596079938s) [noise=true , address=45.91.116.11:4117 , piece_id=QUGUYHEJWEEV733CBC3ONMV7XWDKVH2OFLQWXPQHPKZ7GTH4RLUA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: ecclient: upload failed (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, address:45.91.116.11:4117): protocol: write tcp 188.166.225.170:41292->45.91.116.11:4117: use of closed network connection; write tcp 188.166.225.170:41292->45.91.116.11:4117: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41292->45.91.116.11:4117: use of closed network connection Nov 5 18:52:35.900342116 (151h12m7.596190859s) [noise=true , address=77.90.31.15:4698 , piece_id=7VACL26N7ZCBXIA2I3XL5Q3UJ2BAHDIPXX4VZCQX3S63ABNW625Q, seg_pos={0 0}, upload=133242]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: write tcp 188.166.225.170:41118->77.90.31.15:4698: use of closed network connection; write tcp 188.166.225.170:41118->77.90.31.15:4698: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41118->77.90.31.15:4698: use of closed network connection Nov 5 18:52:35.900382814 (151h12m7.596231567s) [seg_pos={0 0}, upload=133242]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Nov 5 18:52:36.162442033 (151h12m7.858290786s) [upload=133242]: Done uploading segments... Nov 5 18:52:36.162481568 (151h12m7.858330321s) [upload=133242]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 6.038365136s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [............................................X...X...........] duration: 6.219937048s trace: https://jaeger.dev.storj.io/trace/74cf1e30f6d49950 FAIL (2025-11-05 19:01:27 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.219960074s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [..........................................XX...X............] duration: 3.481998883s trace: https://jaeger.dev.storj.io/trace/204ab2570dc82d9c FAIL (2025-11-05 19:01:25 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.48207613s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..........................................X...X.............] duration: 5.83093933s trace: https://jaeger.dev.storj.io/trace/2e576aae48e8f03e FAIL (2025-11-05 19:01:06 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== 5 18:52:06.850167887 (151h11m38.54601663s) [noise=true , address=145.79.227.25:4618 , piece_id=XRUPY5BUPG7BXN2MI6MGYO4N3GW6TZDJE46MZFNRELWANHHG3N5A, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.850747301 (151h11m38.546596054s) [noise=true , address=77.90.31.15:4698 , piece_id=7VACL26N7ZCBXIA2I3XL5Q3UJ2BAHDIPXX4VZCQX3S63ABNW625Q, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.851222605 (151h11m38.547071358s) [noise=true , address=145.79.227.17:4972 , piece_id=H7UHQV7RDLCZUHY2XMXUAD2PX4ZBLWKTM47N6PH7UOGBRXPMWVOA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.851558575 (151h11m38.547407319s) [noise=true , address=145.79.227.22:4509 , piece_id=7RA5NJ72PI3DT4IC2ECVLZ3A2HXXL546ELVDXHZW4OXNN4D36OEA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.851909263 (151h11m38.547758006s) [noise=true , address=145.79.227.26:4654 , piece_id=E5J22KNPQSETSY443QQZXFFKUUJFHWFV22SQAVZ4BFQ4M6YWCNQQ, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.852327540 (151h11m38.548176293s) [noise=true , address=45.91.116.7:4010 , piece_id=EBVGGNQF3N4MTUIATJUWK4QZN7SEOWI6SIAW6N7JZOB5G6H33NEQ, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.852728624 (151h11m38.548577377s) [noise=true , address=145.79.227.9:4366 , piece_id=7ZRECKBQC4S6QLERBZY442WYS4H35XWOZYSGGPHSU7NQ6WL4UNVA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.853978774 (151h11m38.549827528s) [noise=true , address=77.90.31.18:4805 , piece_id=U2U34KK4AJK4A5HGNTHEYCIARLAR4X5NLOVCF63TN3PJUIRMMIHA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.854447296 (151h11m38.550296059s) [noise=true , address=77.90.31.16:4734 , piece_id=HFWVFM46PZFLZT6YNZVIA6HKLVN2PTV2GETZEKX356DMDGQO4HOA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.856523089 (151h11m38.552371842s) [noise=true , address=45.91.116.11:4117 , piece_id=QUGUYHEJWEEV733CBC3ONMV7XWDKVH2OFLQWXPQHPKZ7GTH4RLUA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.859490379 (151h11m38.555339122s) [noise=true , address=77.90.31.13:4628 , piece_id=QFTEHZ5FY7A5DPZZ44TRXGFRHPRNJZJHLQZDGRGOWYFEZYIZT2NQ, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.861350291 (151h11m38.557199034s) [noise=true , address=145.79.227.31:4797 , piece_id=2TFSZNURIAU573QUA5DUS7AJZNY2OXC6TJL3ST4VXIWY2WVB5YJA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.863368665 (151h11m38.559217418s) [noise=true , address=145.79.227.23:4547 , piece_id=K54H437KW5AAYUWOTMDBDP5GJ3ZABFKLAAOSS6WS5G5CROYEHODQ, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.864848794 (151h11m38.560697547s) [noise=true , address=145.79.227.24:4581 , piece_id=TSCVCH2KJEVMD4AVOEIHGTMRCRZ6NZMZA2CTQ6FUO2EIL3JAVYEA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.865644930 (151h11m38.561493683s) [noise=true , address=145.79.227.8:4330 , piece_id=4A47HIUX2MTJ2673XZZCMMF4T235TS4JWMNYJEVHHCKAHQX6APMA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:08.560504137 (151h11m40.25635289s) [upload=133242]: Next returned nil segment Nov 5 18:52:08.560522693 (151h11m40.256371446s) [upload=133242]: Waiting for error group managing segments... Nov 5 18:52:08.850333179 (151h11m40.546181932s) [noise=true , address=145.79.227.17:4972 , piece_id=H7UHQV7RDLCZUHY2XMXUAD2PX4ZBLWKTM47N6PH7UOGBRXPMWVOA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.853517822 (151h11m40.549366575s) [noise=true , address=145.79.227.22:4509 , piece_id=7RA5NJ72PI3DT4IC2ECVLZ3A2HXXL546ELVDXHZW4OXNN4D36OEA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.855547508 (151h11m40.551396261s) [noise=true , address=145.79.227.9:4366 , piece_id=7ZRECKBQC4S6QLERBZY442WYS4H35XWOZYSGGPHSU7NQ6WL4UNVA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.856103106 (151h11m40.551951859s) [noise=true , address=145.79.227.25:4618 , piece_id=XRUPY5BUPG7BXN2MI6MGYO4N3GW6TZDJE46MZFNRELWANHHG3N5A, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.860417321 (151h11m40.556266064s) [noise=true , address=145.79.227.8:4330 , piece_id=4A47HIUX2MTJ2673XZZCMMF4T235TS4JWMNYJEVHHCKAHQX6APMA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.865602796 (151h11m40.561451549s) [noise=true , address=145.79.227.23:4547 , piece_id=K54H437KW5AAYUWOTMDBDP5GJ3ZABFKLAAOSS6WS5G5CROYEHODQ, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.879393757 (151h11m40.57524251s) [noise=true , address=145.79.227.31:4797 , piece_id=2TFSZNURIAU573QUA5DUS7AJZNY2OXC6TJL3ST4VXIWY2WVB5YJA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.905541449 (151h11m40.601390202s) [noise=true , address=145.79.227.26:4654 , piece_id=E5J22KNPQSETSY443QQZXFFKUUJFHWFV22SQAVZ4BFQ4M6YWCNQQ, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.929018836 (151h11m40.624867589s) [noise=true , address=145.79.227.24:4581 , piece_id=TSCVCH2KJEVMD4AVOEIHGTMRCRZ6NZMZA2CTQ6FUO2EIL3JAVYEA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:35.899554917 (151h12m7.59540368s) [noise=true , address=77.90.31.17:4771 , piece_id=KVFE7VYW3RZJ25BWSH4BIRRQCMWDYOIK7EJ62XVREGN26YN7XYMQ, seg_pos={0 0}, upload=133242]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: write tcp 188.166.225.170:60676->77.90.31.17:4771: use of closed network connection; write tcp 188.166.225.170:60676->77.90.31.17:4771: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60676->77.90.31.17:4771: use of closed network connection Nov 5 18:52:35.899727506 (151h12m7.595576259s) [noise=true , address=77.90.31.13:4628 , piece_id=QFTEHZ5FY7A5DPZZ44TRXGFRHPRNJZJHLQZDGRGOWYFEZYIZT2NQ, seg_pos={0 0}, upload=133242]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: write tcp 188.166.225.170:42026->77.90.31.13:4628: use of closed network connection; write tcp 188.166.225.170:42026->77.90.31.13:4628: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42026->77.90.31.13:4628: use of closed network connection Nov 5 18:52:35.899860439 (151h12m7.595709192s) [noise=true , address=45.91.116.7:4010 , piece_id=EBVGGNQF3N4MTUIATJUWK4QZN7SEOWI6SIAW6N7JZOB5G6H33NEQ, seg_pos={0 0}, upload=133242]: Done uploading piece. err: ecclient: upload failed (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, address:45.91.116.7:4010): protocol: write tcp 188.166.225.170:40036->45.91.116.7:4010: use of closed network connection; write tcp 188.166.225.170:40036->45.91.116.7:4010: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40036->45.91.116.7:4010: use of closed network connection Nov 5 18:52:35.899923829 (151h12m7.595772572s) [noise=true , address=77.90.31.18:4805 , piece_id=U2U34KK4AJK4A5HGNTHEYCIARLAR4X5NLOVCF63TN3PJUIRMMIHA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: write tcp 188.166.225.170:60758->77.90.31.18:4805: use of closed network connection; write tcp 188.166.225.170:60758->77.90.31.18:4805: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60758->77.90.31.18:4805: use of closed network connection Nov 5 18:52:35.900052084 (151h12m7.595900837s) [noise=true , address=77.90.31.16:4734 , piece_id=HFWVFM46PZFLZT6YNZVIA6HKLVN2PTV2GETZEKX356DMDGQO4HOA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: write tcp 188.166.225.170:50728->77.90.31.16:4734: use of closed network connection; write tcp 188.166.225.170:50728->77.90.31.16:4734: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50728->77.90.31.16:4734: use of closed network connection Nov 5 18:52:35.900231175 (151h12m7.596079938s) [noise=true , address=45.91.116.11:4117 , piece_id=QUGUYHEJWEEV733CBC3ONMV7XWDKVH2OFLQWXPQHPKZ7GTH4RLUA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: ecclient: upload failed (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, address:45.91.116.11:4117): protocol: write tcp 188.166.225.170:41292->45.91.116.11:4117: use of closed network connection; write tcp 188.166.225.170:41292->45.91.116.11:4117: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41292->45.91.116.11:4117: use of closed network connection Nov 5 18:52:35.900342116 (151h12m7.596190859s) [noise=true , address=77.90.31.15:4698 , piece_id=7VACL26N7ZCBXIA2I3XL5Q3UJ2BAHDIPXX4VZCQX3S63ABNW625Q, seg_pos={0 0}, upload=133242]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: write tcp 188.166.225.170:41118->77.90.31.15:4698: use of closed network connection; write tcp 188.166.225.170:41118->77.90.31.15:4698: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41118->77.90.31.15:4698: use of closed network connection Nov 5 18:52:35.900382814 (151h12m7.596231567s) [seg_pos={0 0}, upload=133242]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Nov 5 18:52:36.162442033 (151h12m7.858290786s) [upload=133242]: Done uploading segments... Nov 5 18:52:36.162481568 (151h12m7.858330321s) [upload=133242]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 5.830971611s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [..........................................X..X..............] duration: 5.992046023s trace: https://jaeger.dev.storj.io/trace/3e2ae8884260b157 FAIL (2025-11-05 19:01:06 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.992085809s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [...........................................X..X.............] duration: 5.751179651s trace: https://jaeger.dev.storj.io/trace/3e2ae8884260b157 FAIL (2025-11-05 19:00:32 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.751293278s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...........................................X...X............] duration: 6.160145116s trace: https://jaeger.dev.storj.io/trace/2e576aae48e8f03e FAIL (2025-11-05 19:00:13 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== 5 18:52:06.850167887 (151h11m38.54601663s) [noise=true , address=145.79.227.25:4618 , piece_id=XRUPY5BUPG7BXN2MI6MGYO4N3GW6TZDJE46MZFNRELWANHHG3N5A, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.850747301 (151h11m38.546596054s) [noise=true , address=77.90.31.15:4698 , piece_id=7VACL26N7ZCBXIA2I3XL5Q3UJ2BAHDIPXX4VZCQX3S63ABNW625Q, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.851222605 (151h11m38.547071358s) [noise=true , address=145.79.227.17:4972 , piece_id=H7UHQV7RDLCZUHY2XMXUAD2PX4ZBLWKTM47N6PH7UOGBRXPMWVOA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.851558575 (151h11m38.547407319s) [noise=true , address=145.79.227.22:4509 , piece_id=7RA5NJ72PI3DT4IC2ECVLZ3A2HXXL546ELVDXHZW4OXNN4D36OEA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.851909263 (151h11m38.547758006s) [noise=true , address=145.79.227.26:4654 , piece_id=E5J22KNPQSETSY443QQZXFFKUUJFHWFV22SQAVZ4BFQ4M6YWCNQQ, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.852327540 (151h11m38.548176293s) [noise=true , address=45.91.116.7:4010 , piece_id=EBVGGNQF3N4MTUIATJUWK4QZN7SEOWI6SIAW6N7JZOB5G6H33NEQ, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.852728624 (151h11m38.548577377s) [noise=true , address=145.79.227.9:4366 , piece_id=7ZRECKBQC4S6QLERBZY442WYS4H35XWOZYSGGPHSU7NQ6WL4UNVA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.853978774 (151h11m38.549827528s) [noise=true , address=77.90.31.18:4805 , piece_id=U2U34KK4AJK4A5HGNTHEYCIARLAR4X5NLOVCF63TN3PJUIRMMIHA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.854447296 (151h11m38.550296059s) [noise=true , address=77.90.31.16:4734 , piece_id=HFWVFM46PZFLZT6YNZVIA6HKLVN2PTV2GETZEKX356DMDGQO4HOA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.856523089 (151h11m38.552371842s) [noise=true , address=45.91.116.11:4117 , piece_id=QUGUYHEJWEEV733CBC3ONMV7XWDKVH2OFLQWXPQHPKZ7GTH4RLUA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.859490379 (151h11m38.555339122s) [noise=true , address=77.90.31.13:4628 , piece_id=QFTEHZ5FY7A5DPZZ44TRXGFRHPRNJZJHLQZDGRGOWYFEZYIZT2NQ, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.861350291 (151h11m38.557199034s) [noise=true , address=145.79.227.31:4797 , piece_id=2TFSZNURIAU573QUA5DUS7AJZNY2OXC6TJL3ST4VXIWY2WVB5YJA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.863368665 (151h11m38.559217418s) [noise=true , address=145.79.227.23:4547 , piece_id=K54H437KW5AAYUWOTMDBDP5GJ3ZABFKLAAOSS6WS5G5CROYEHODQ, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.864848794 (151h11m38.560697547s) [noise=true , address=145.79.227.24:4581 , piece_id=TSCVCH2KJEVMD4AVOEIHGTMRCRZ6NZMZA2CTQ6FUO2EIL3JAVYEA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.865644930 (151h11m38.561493683s) [noise=true , address=145.79.227.8:4330 , piece_id=4A47HIUX2MTJ2673XZZCMMF4T235TS4JWMNYJEVHHCKAHQX6APMA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:08.560504137 (151h11m40.25635289s) [upload=133242]: Next returned nil segment Nov 5 18:52:08.560522693 (151h11m40.256371446s) [upload=133242]: Waiting for error group managing segments... Nov 5 18:52:08.850333179 (151h11m40.546181932s) [noise=true , address=145.79.227.17:4972 , piece_id=H7UHQV7RDLCZUHY2XMXUAD2PX4ZBLWKTM47N6PH7UOGBRXPMWVOA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.853517822 (151h11m40.549366575s) [noise=true , address=145.79.227.22:4509 , piece_id=7RA5NJ72PI3DT4IC2ECVLZ3A2HXXL546ELVDXHZW4OXNN4D36OEA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.855547508 (151h11m40.551396261s) [noise=true , address=145.79.227.9:4366 , piece_id=7ZRECKBQC4S6QLERBZY442WYS4H35XWOZYSGGPHSU7NQ6WL4UNVA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.856103106 (151h11m40.551951859s) [noise=true , address=145.79.227.25:4618 , piece_id=XRUPY5BUPG7BXN2MI6MGYO4N3GW6TZDJE46MZFNRELWANHHG3N5A, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.860417321 (151h11m40.556266064s) [noise=true , address=145.79.227.8:4330 , piece_id=4A47HIUX2MTJ2673XZZCMMF4T235TS4JWMNYJEVHHCKAHQX6APMA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.865602796 (151h11m40.561451549s) [noise=true , address=145.79.227.23:4547 , piece_id=K54H437KW5AAYUWOTMDBDP5GJ3ZABFKLAAOSS6WS5G5CROYEHODQ, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.879393757 (151h11m40.57524251s) [noise=true , address=145.79.227.31:4797 , piece_id=2TFSZNURIAU573QUA5DUS7AJZNY2OXC6TJL3ST4VXIWY2WVB5YJA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.905541449 (151h11m40.601390202s) [noise=true , address=145.79.227.26:4654 , piece_id=E5J22KNPQSETSY443QQZXFFKUUJFHWFV22SQAVZ4BFQ4M6YWCNQQ, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.929018836 (151h11m40.624867589s) [noise=true , address=145.79.227.24:4581 , piece_id=TSCVCH2KJEVMD4AVOEIHGTMRCRZ6NZMZA2CTQ6FUO2EIL3JAVYEA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:35.899554917 (151h12m7.59540368s) [noise=true , address=77.90.31.17:4771 , piece_id=KVFE7VYW3RZJ25BWSH4BIRRQCMWDYOIK7EJ62XVREGN26YN7XYMQ, seg_pos={0 0}, upload=133242]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: write tcp 188.166.225.170:60676->77.90.31.17:4771: use of closed network connection; write tcp 188.166.225.170:60676->77.90.31.17:4771: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60676->77.90.31.17:4771: use of closed network connection Nov 5 18:52:35.899727506 (151h12m7.595576259s) [noise=true , address=77.90.31.13:4628 , piece_id=QFTEHZ5FY7A5DPZZ44TRXGFRHPRNJZJHLQZDGRGOWYFEZYIZT2NQ, seg_pos={0 0}, upload=133242]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: write tcp 188.166.225.170:42026->77.90.31.13:4628: use of closed network connection; write tcp 188.166.225.170:42026->77.90.31.13:4628: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42026->77.90.31.13:4628: use of closed network connection Nov 5 18:52:35.899860439 (151h12m7.595709192s) [noise=true , address=45.91.116.7:4010 , piece_id=EBVGGNQF3N4MTUIATJUWK4QZN7SEOWI6SIAW6N7JZOB5G6H33NEQ, seg_pos={0 0}, upload=133242]: Done uploading piece. err: ecclient: upload failed (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, address:45.91.116.7:4010): protocol: write tcp 188.166.225.170:40036->45.91.116.7:4010: use of closed network connection; write tcp 188.166.225.170:40036->45.91.116.7:4010: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40036->45.91.116.7:4010: use of closed network connection Nov 5 18:52:35.899923829 (151h12m7.595772572s) [noise=true , address=77.90.31.18:4805 , piece_id=U2U34KK4AJK4A5HGNTHEYCIARLAR4X5NLOVCF63TN3PJUIRMMIHA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: write tcp 188.166.225.170:60758->77.90.31.18:4805: use of closed network connection; write tcp 188.166.225.170:60758->77.90.31.18:4805: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60758->77.90.31.18:4805: use of closed network connection Nov 5 18:52:35.900052084 (151h12m7.595900837s) [noise=true , address=77.90.31.16:4734 , piece_id=HFWVFM46PZFLZT6YNZVIA6HKLVN2PTV2GETZEKX356DMDGQO4HOA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: write tcp 188.166.225.170:50728->77.90.31.16:4734: use of closed network connection; write tcp 188.166.225.170:50728->77.90.31.16:4734: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50728->77.90.31.16:4734: use of closed network connection Nov 5 18:52:35.900231175 (151h12m7.596079938s) [noise=true , address=45.91.116.11:4117 , piece_id=QUGUYHEJWEEV733CBC3ONMV7XWDKVH2OFLQWXPQHPKZ7GTH4RLUA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: ecclient: upload failed (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, address:45.91.116.11:4117): protocol: write tcp 188.166.225.170:41292->45.91.116.11:4117: use of closed network connection; write tcp 188.166.225.170:41292->45.91.116.11:4117: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41292->45.91.116.11:4117: use of closed network connection Nov 5 18:52:35.900342116 (151h12m7.596190859s) [noise=true , address=77.90.31.15:4698 , piece_id=7VACL26N7ZCBXIA2I3XL5Q3UJ2BAHDIPXX4VZCQX3S63ABNW625Q, seg_pos={0 0}, upload=133242]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: write tcp 188.166.225.170:41118->77.90.31.15:4698: use of closed network connection; write tcp 188.166.225.170:41118->77.90.31.15:4698: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41118->77.90.31.15:4698: use of closed network connection Nov 5 18:52:35.900382814 (151h12m7.596231567s) [seg_pos={0 0}, upload=133242]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Nov 5 18:52:36.162442033 (151h12m7.858290786s) [upload=133242]: Done uploading segments... Nov 5 18:52:36.162481568 (151h12m7.858330321s) [upload=133242]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 6.160167478s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.............................................X...X..........] duration: 6.681906874s trace: https://jaeger.dev.storj.io/trace/74cf1e30f6d49950 FAIL (2025-11-05 19:00:12 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.681942624s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [...........................................XX...X...........] duration: 3.589132997s trace: https://jaeger.dev.storj.io/trace/204ab2570dc82d9c FAIL (2025-11-05 19:00:12 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.589164547s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [............................................X..X............] duration: 5.693019128s trace: https://jaeger.dev.storj.io/trace/3e2ae8884260b157 FAIL (2025-11-05 18:59:58 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.693063263s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..............................................X...X.........] duration: 6.207929666s trace: https://jaeger.dev.storj.io/trace/74cf1e30f6d49950 FAIL (2025-11-05 18:59:51 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.207969222s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [............................................XX...X..........] duration: 3.429523603s trace: https://jaeger.dev.storj.io/trace/204ab2570dc82d9c FAIL (2025-11-05 18:59:38 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.429552357s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...............................................X...X........] duration: 6.345470171s trace: https://jaeger.dev.storj.io/trace/74cf1e30f6d49950 FAIL (2025-11-05 18:59:36 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.345498094s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [............................................X...X...........] duration: 5.973901377s trace: https://jaeger.dev.storj.io/trace/2e576aae48e8f03e FAIL (2025-11-05 18:59:31 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== 5 18:52:06.850167887 (151h11m38.54601663s) [noise=true , address=145.79.227.25:4618 , piece_id=XRUPY5BUPG7BXN2MI6MGYO4N3GW6TZDJE46MZFNRELWANHHG3N5A, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.850747301 (151h11m38.546596054s) [noise=true , address=77.90.31.15:4698 , piece_id=7VACL26N7ZCBXIA2I3XL5Q3UJ2BAHDIPXX4VZCQX3S63ABNW625Q, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.851222605 (151h11m38.547071358s) [noise=true , address=145.79.227.17:4972 , piece_id=H7UHQV7RDLCZUHY2XMXUAD2PX4ZBLWKTM47N6PH7UOGBRXPMWVOA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.851558575 (151h11m38.547407319s) [noise=true , address=145.79.227.22:4509 , piece_id=7RA5NJ72PI3DT4IC2ECVLZ3A2HXXL546ELVDXHZW4OXNN4D36OEA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.851909263 (151h11m38.547758006s) [noise=true , address=145.79.227.26:4654 , piece_id=E5J22KNPQSETSY443QQZXFFKUUJFHWFV22SQAVZ4BFQ4M6YWCNQQ, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.852327540 (151h11m38.548176293s) [noise=true , address=45.91.116.7:4010 , piece_id=EBVGGNQF3N4MTUIATJUWK4QZN7SEOWI6SIAW6N7JZOB5G6H33NEQ, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.852728624 (151h11m38.548577377s) [noise=true , address=145.79.227.9:4366 , piece_id=7ZRECKBQC4S6QLERBZY442WYS4H35XWOZYSGGPHSU7NQ6WL4UNVA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.853978774 (151h11m38.549827528s) [noise=true , address=77.90.31.18:4805 , piece_id=U2U34KK4AJK4A5HGNTHEYCIARLAR4X5NLOVCF63TN3PJUIRMMIHA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.854447296 (151h11m38.550296059s) [noise=true , address=77.90.31.16:4734 , piece_id=HFWVFM46PZFLZT6YNZVIA6HKLVN2PTV2GETZEKX356DMDGQO4HOA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.856523089 (151h11m38.552371842s) [noise=true , address=45.91.116.11:4117 , piece_id=QUGUYHEJWEEV733CBC3ONMV7XWDKVH2OFLQWXPQHPKZ7GTH4RLUA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.859490379 (151h11m38.555339122s) [noise=true , address=77.90.31.13:4628 , piece_id=QFTEHZ5FY7A5DPZZ44TRXGFRHPRNJZJHLQZDGRGOWYFEZYIZT2NQ, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.861350291 (151h11m38.557199034s) [noise=true , address=145.79.227.31:4797 , piece_id=2TFSZNURIAU573QUA5DUS7AJZNY2OXC6TJL3ST4VXIWY2WVB5YJA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.863368665 (151h11m38.559217418s) [noise=true , address=145.79.227.23:4547 , piece_id=K54H437KW5AAYUWOTMDBDP5GJ3ZABFKLAAOSS6WS5G5CROYEHODQ, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.864848794 (151h11m38.560697547s) [noise=true , address=145.79.227.24:4581 , piece_id=TSCVCH2KJEVMD4AVOEIHGTMRCRZ6NZMZA2CTQ6FUO2EIL3JAVYEA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.865644930 (151h11m38.561493683s) [noise=true , address=145.79.227.8:4330 , piece_id=4A47HIUX2MTJ2673XZZCMMF4T235TS4JWMNYJEVHHCKAHQX6APMA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:08.560504137 (151h11m40.25635289s) [upload=133242]: Next returned nil segment Nov 5 18:52:08.560522693 (151h11m40.256371446s) [upload=133242]: Waiting for error group managing segments... Nov 5 18:52:08.850333179 (151h11m40.546181932s) [noise=true , address=145.79.227.17:4972 , piece_id=H7UHQV7RDLCZUHY2XMXUAD2PX4ZBLWKTM47N6PH7UOGBRXPMWVOA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.853517822 (151h11m40.549366575s) [noise=true , address=145.79.227.22:4509 , piece_id=7RA5NJ72PI3DT4IC2ECVLZ3A2HXXL546ELVDXHZW4OXNN4D36OEA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.855547508 (151h11m40.551396261s) [noise=true , address=145.79.227.9:4366 , piece_id=7ZRECKBQC4S6QLERBZY442WYS4H35XWOZYSGGPHSU7NQ6WL4UNVA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.856103106 (151h11m40.551951859s) [noise=true , address=145.79.227.25:4618 , piece_id=XRUPY5BUPG7BXN2MI6MGYO4N3GW6TZDJE46MZFNRELWANHHG3N5A, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.860417321 (151h11m40.556266064s) [noise=true , address=145.79.227.8:4330 , piece_id=4A47HIUX2MTJ2673XZZCMMF4T235TS4JWMNYJEVHHCKAHQX6APMA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.865602796 (151h11m40.561451549s) [noise=true , address=145.79.227.23:4547 , piece_id=K54H437KW5AAYUWOTMDBDP5GJ3ZABFKLAAOSS6WS5G5CROYEHODQ, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.879393757 (151h11m40.57524251s) [noise=true , address=145.79.227.31:4797 , piece_id=2TFSZNURIAU573QUA5DUS7AJZNY2OXC6TJL3ST4VXIWY2WVB5YJA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.905541449 (151h11m40.601390202s) [noise=true , address=145.79.227.26:4654 , piece_id=E5J22KNPQSETSY443QQZXFFKUUJFHWFV22SQAVZ4BFQ4M6YWCNQQ, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.929018836 (151h11m40.624867589s) [noise=true , address=145.79.227.24:4581 , piece_id=TSCVCH2KJEVMD4AVOEIHGTMRCRZ6NZMZA2CTQ6FUO2EIL3JAVYEA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:35.899554917 (151h12m7.59540368s) [noise=true , address=77.90.31.17:4771 , piece_id=KVFE7VYW3RZJ25BWSH4BIRRQCMWDYOIK7EJ62XVREGN26YN7XYMQ, seg_pos={0 0}, upload=133242]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: write tcp 188.166.225.170:60676->77.90.31.17:4771: use of closed network connection; write tcp 188.166.225.170:60676->77.90.31.17:4771: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60676->77.90.31.17:4771: use of closed network connection Nov 5 18:52:35.899727506 (151h12m7.595576259s) [noise=true , address=77.90.31.13:4628 , piece_id=QFTEHZ5FY7A5DPZZ44TRXGFRHPRNJZJHLQZDGRGOWYFEZYIZT2NQ, seg_pos={0 0}, upload=133242]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: write tcp 188.166.225.170:42026->77.90.31.13:4628: use of closed network connection; write tcp 188.166.225.170:42026->77.90.31.13:4628: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42026->77.90.31.13:4628: use of closed network connection Nov 5 18:52:35.899860439 (151h12m7.595709192s) [noise=true , address=45.91.116.7:4010 , piece_id=EBVGGNQF3N4MTUIATJUWK4QZN7SEOWI6SIAW6N7JZOB5G6H33NEQ, seg_pos={0 0}, upload=133242]: Done uploading piece. err: ecclient: upload failed (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, address:45.91.116.7:4010): protocol: write tcp 188.166.225.170:40036->45.91.116.7:4010: use of closed network connection; write tcp 188.166.225.170:40036->45.91.116.7:4010: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40036->45.91.116.7:4010: use of closed network connection Nov 5 18:52:35.899923829 (151h12m7.595772572s) [noise=true , address=77.90.31.18:4805 , piece_id=U2U34KK4AJK4A5HGNTHEYCIARLAR4X5NLOVCF63TN3PJUIRMMIHA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: write tcp 188.166.225.170:60758->77.90.31.18:4805: use of closed network connection; write tcp 188.166.225.170:60758->77.90.31.18:4805: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60758->77.90.31.18:4805: use of closed network connection Nov 5 18:52:35.900052084 (151h12m7.595900837s) [noise=true , address=77.90.31.16:4734 , piece_id=HFWVFM46PZFLZT6YNZVIA6HKLVN2PTV2GETZEKX356DMDGQO4HOA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: write tcp 188.166.225.170:50728->77.90.31.16:4734: use of closed network connection; write tcp 188.166.225.170:50728->77.90.31.16:4734: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50728->77.90.31.16:4734: use of closed network connection Nov 5 18:52:35.900231175 (151h12m7.596079938s) [noise=true , address=45.91.116.11:4117 , piece_id=QUGUYHEJWEEV733CBC3ONMV7XWDKVH2OFLQWXPQHPKZ7GTH4RLUA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: ecclient: upload failed (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, address:45.91.116.11:4117): protocol: write tcp 188.166.225.170:41292->45.91.116.11:4117: use of closed network connection; write tcp 188.166.225.170:41292->45.91.116.11:4117: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41292->45.91.116.11:4117: use of closed network connection Nov 5 18:52:35.900342116 (151h12m7.596190859s) [noise=true , address=77.90.31.15:4698 , piece_id=7VACL26N7ZCBXIA2I3XL5Q3UJ2BAHDIPXX4VZCQX3S63ABNW625Q, seg_pos={0 0}, upload=133242]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: write tcp 188.166.225.170:41118->77.90.31.15:4698: use of closed network connection; write tcp 188.166.225.170:41118->77.90.31.15:4698: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41118->77.90.31.15:4698: use of closed network connection Nov 5 18:52:35.900382814 (151h12m7.596231567s) [seg_pos={0 0}, upload=133242]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Nov 5 18:52:36.162442033 (151h12m7.858290786s) [upload=133242]: Done uploading segments... Nov 5 18:52:36.162481568 (151h12m7.858330321s) [upload=133242]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 5.973931415s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.............................................X..X...........] duration: 5.687331063s trace: https://jaeger.dev.storj.io/trace/3e2ae8884260b157 FAIL (2025-11-05 18:59:30 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.687360259s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.............................................X...X..........] duration: 5.987762504s trace: https://jaeger.dev.storj.io/trace/2e576aae48e8f03e FAIL (2025-11-05 18:59:10 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== 5 18:52:06.850167887 (151h11m38.54601663s) [noise=true , address=145.79.227.25:4618 , piece_id=XRUPY5BUPG7BXN2MI6MGYO4N3GW6TZDJE46MZFNRELWANHHG3N5A, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.850747301 (151h11m38.546596054s) [noise=true , address=77.90.31.15:4698 , piece_id=7VACL26N7ZCBXIA2I3XL5Q3UJ2BAHDIPXX4VZCQX3S63ABNW625Q, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.851222605 (151h11m38.547071358s) [noise=true , address=145.79.227.17:4972 , piece_id=H7UHQV7RDLCZUHY2XMXUAD2PX4ZBLWKTM47N6PH7UOGBRXPMWVOA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.851558575 (151h11m38.547407319s) [noise=true , address=145.79.227.22:4509 , piece_id=7RA5NJ72PI3DT4IC2ECVLZ3A2HXXL546ELVDXHZW4OXNN4D36OEA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.851909263 (151h11m38.547758006s) [noise=true , address=145.79.227.26:4654 , piece_id=E5J22KNPQSETSY443QQZXFFKUUJFHWFV22SQAVZ4BFQ4M6YWCNQQ, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.852327540 (151h11m38.548176293s) [noise=true , address=45.91.116.7:4010 , piece_id=EBVGGNQF3N4MTUIATJUWK4QZN7SEOWI6SIAW6N7JZOB5G6H33NEQ, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.852728624 (151h11m38.548577377s) [noise=true , address=145.79.227.9:4366 , piece_id=7ZRECKBQC4S6QLERBZY442WYS4H35XWOZYSGGPHSU7NQ6WL4UNVA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.853978774 (151h11m38.549827528s) [noise=true , address=77.90.31.18:4805 , piece_id=U2U34KK4AJK4A5HGNTHEYCIARLAR4X5NLOVCF63TN3PJUIRMMIHA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.854447296 (151h11m38.550296059s) [noise=true , address=77.90.31.16:4734 , piece_id=HFWVFM46PZFLZT6YNZVIA6HKLVN2PTV2GETZEKX356DMDGQO4HOA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.856523089 (151h11m38.552371842s) [noise=true , address=45.91.116.11:4117 , piece_id=QUGUYHEJWEEV733CBC3ONMV7XWDKVH2OFLQWXPQHPKZ7GTH4RLUA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.859490379 (151h11m38.555339122s) [noise=true , address=77.90.31.13:4628 , piece_id=QFTEHZ5FY7A5DPZZ44TRXGFRHPRNJZJHLQZDGRGOWYFEZYIZT2NQ, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.861350291 (151h11m38.557199034s) [noise=true , address=145.79.227.31:4797 , piece_id=2TFSZNURIAU573QUA5DUS7AJZNY2OXC6TJL3ST4VXIWY2WVB5YJA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.863368665 (151h11m38.559217418s) [noise=true , address=145.79.227.23:4547 , piece_id=K54H437KW5AAYUWOTMDBDP5GJ3ZABFKLAAOSS6WS5G5CROYEHODQ, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.864848794 (151h11m38.560697547s) [noise=true , address=145.79.227.24:4581 , piece_id=TSCVCH2KJEVMD4AVOEIHGTMRCRZ6NZMZA2CTQ6FUO2EIL3JAVYEA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.865644930 (151h11m38.561493683s) [noise=true , address=145.79.227.8:4330 , piece_id=4A47HIUX2MTJ2673XZZCMMF4T235TS4JWMNYJEVHHCKAHQX6APMA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:08.560504137 (151h11m40.25635289s) [upload=133242]: Next returned nil segment Nov 5 18:52:08.560522693 (151h11m40.256371446s) [upload=133242]: Waiting for error group managing segments... Nov 5 18:52:08.850333179 (151h11m40.546181932s) [noise=true , address=145.79.227.17:4972 , piece_id=H7UHQV7RDLCZUHY2XMXUAD2PX4ZBLWKTM47N6PH7UOGBRXPMWVOA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.853517822 (151h11m40.549366575s) [noise=true , address=145.79.227.22:4509 , piece_id=7RA5NJ72PI3DT4IC2ECVLZ3A2HXXL546ELVDXHZW4OXNN4D36OEA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.855547508 (151h11m40.551396261s) [noise=true , address=145.79.227.9:4366 , piece_id=7ZRECKBQC4S6QLERBZY442WYS4H35XWOZYSGGPHSU7NQ6WL4UNVA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.856103106 (151h11m40.551951859s) [noise=true , address=145.79.227.25:4618 , piece_id=XRUPY5BUPG7BXN2MI6MGYO4N3GW6TZDJE46MZFNRELWANHHG3N5A, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.860417321 (151h11m40.556266064s) [noise=true , address=145.79.227.8:4330 , piece_id=4A47HIUX2MTJ2673XZZCMMF4T235TS4JWMNYJEVHHCKAHQX6APMA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.865602796 (151h11m40.561451549s) [noise=true , address=145.79.227.23:4547 , piece_id=K54H437KW5AAYUWOTMDBDP5GJ3ZABFKLAAOSS6WS5G5CROYEHODQ, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.879393757 (151h11m40.57524251s) [noise=true , address=145.79.227.31:4797 , piece_id=2TFSZNURIAU573QUA5DUS7AJZNY2OXC6TJL3ST4VXIWY2WVB5YJA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.905541449 (151h11m40.601390202s) [noise=true , address=145.79.227.26:4654 , piece_id=E5J22KNPQSETSY443QQZXFFKUUJFHWFV22SQAVZ4BFQ4M6YWCNQQ, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.929018836 (151h11m40.624867589s) [noise=true , address=145.79.227.24:4581 , piece_id=TSCVCH2KJEVMD4AVOEIHGTMRCRZ6NZMZA2CTQ6FUO2EIL3JAVYEA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:35.899554917 (151h12m7.59540368s) [noise=true , address=77.90.31.17:4771 , piece_id=KVFE7VYW3RZJ25BWSH4BIRRQCMWDYOIK7EJ62XVREGN26YN7XYMQ, seg_pos={0 0}, upload=133242]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: write tcp 188.166.225.170:60676->77.90.31.17:4771: use of closed network connection; write tcp 188.166.225.170:60676->77.90.31.17:4771: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60676->77.90.31.17:4771: use of closed network connection Nov 5 18:52:35.899727506 (151h12m7.595576259s) [noise=true , address=77.90.31.13:4628 , piece_id=QFTEHZ5FY7A5DPZZ44TRXGFRHPRNJZJHLQZDGRGOWYFEZYIZT2NQ, seg_pos={0 0}, upload=133242]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: write tcp 188.166.225.170:42026->77.90.31.13:4628: use of closed network connection; write tcp 188.166.225.170:42026->77.90.31.13:4628: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42026->77.90.31.13:4628: use of closed network connection Nov 5 18:52:35.899860439 (151h12m7.595709192s) [noise=true , address=45.91.116.7:4010 , piece_id=EBVGGNQF3N4MTUIATJUWK4QZN7SEOWI6SIAW6N7JZOB5G6H33NEQ, seg_pos={0 0}, upload=133242]: Done uploading piece. err: ecclient: upload failed (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, address:45.91.116.7:4010): protocol: write tcp 188.166.225.170:40036->45.91.116.7:4010: use of closed network connection; write tcp 188.166.225.170:40036->45.91.116.7:4010: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40036->45.91.116.7:4010: use of closed network connection Nov 5 18:52:35.899923829 (151h12m7.595772572s) [noise=true , address=77.90.31.18:4805 , piece_id=U2U34KK4AJK4A5HGNTHEYCIARLAR4X5NLOVCF63TN3PJUIRMMIHA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: write tcp 188.166.225.170:60758->77.90.31.18:4805: use of closed network connection; write tcp 188.166.225.170:60758->77.90.31.18:4805: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60758->77.90.31.18:4805: use of closed network connection Nov 5 18:52:35.900052084 (151h12m7.595900837s) [noise=true , address=77.90.31.16:4734 , piece_id=HFWVFM46PZFLZT6YNZVIA6HKLVN2PTV2GETZEKX356DMDGQO4HOA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: write tcp 188.166.225.170:50728->77.90.31.16:4734: use of closed network connection; write tcp 188.166.225.170:50728->77.90.31.16:4734: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50728->77.90.31.16:4734: use of closed network connection Nov 5 18:52:35.900231175 (151h12m7.596079938s) [noise=true , address=45.91.116.11:4117 , piece_id=QUGUYHEJWEEV733CBC3ONMV7XWDKVH2OFLQWXPQHPKZ7GTH4RLUA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: ecclient: upload failed (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, address:45.91.116.11:4117): protocol: write tcp 188.166.225.170:41292->45.91.116.11:4117: use of closed network connection; write tcp 188.166.225.170:41292->45.91.116.11:4117: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41292->45.91.116.11:4117: use of closed network connection Nov 5 18:52:35.900342116 (151h12m7.596190859s) [noise=true , address=77.90.31.15:4698 , piece_id=7VACL26N7ZCBXIA2I3XL5Q3UJ2BAHDIPXX4VZCQX3S63ABNW625Q, seg_pos={0 0}, upload=133242]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: write tcp 188.166.225.170:41118->77.90.31.15:4698: use of closed network connection; write tcp 188.166.225.170:41118->77.90.31.15:4698: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41118->77.90.31.15:4698: use of closed network connection Nov 5 18:52:35.900382814 (151h12m7.596231567s) [seg_pos={0 0}, upload=133242]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Nov 5 18:52:36.162442033 (151h12m7.858290786s) [upload=133242]: Done uploading segments... Nov 5 18:52:36.162481568 (151h12m7.858330321s) [upload=133242]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 5.987794946s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [..............................................X..X..........] duration: 5.778065237s trace: https://jaeger.dev.storj.io/trace/3e2ae8884260b157 FAIL (2025-11-05 18:59:10 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.778099853s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [.............................................XX...X.........] duration: 3.623551925s trace: https://jaeger.dev.storj.io/trace/204ab2570dc82d9c FAIL (2025-11-05 18:58:55 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.62357575s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..............................................X...X.........] duration: 5.923150633s trace: https://jaeger.dev.storj.io/trace/2e576aae48e8f03e FAIL (2025-11-05 18:58:48 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== 5 18:52:06.850167887 (151h11m38.54601663s) [noise=true , address=145.79.227.25:4618 , piece_id=XRUPY5BUPG7BXN2MI6MGYO4N3GW6TZDJE46MZFNRELWANHHG3N5A, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.850747301 (151h11m38.546596054s) [noise=true , address=77.90.31.15:4698 , piece_id=7VACL26N7ZCBXIA2I3XL5Q3UJ2BAHDIPXX4VZCQX3S63ABNW625Q, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.851222605 (151h11m38.547071358s) [noise=true , address=145.79.227.17:4972 , piece_id=H7UHQV7RDLCZUHY2XMXUAD2PX4ZBLWKTM47N6PH7UOGBRXPMWVOA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.851558575 (151h11m38.547407319s) [noise=true , address=145.79.227.22:4509 , piece_id=7RA5NJ72PI3DT4IC2ECVLZ3A2HXXL546ELVDXHZW4OXNN4D36OEA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.851909263 (151h11m38.547758006s) [noise=true , address=145.79.227.26:4654 , piece_id=E5J22KNPQSETSY443QQZXFFKUUJFHWFV22SQAVZ4BFQ4M6YWCNQQ, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.852327540 (151h11m38.548176293s) [noise=true , address=45.91.116.7:4010 , piece_id=EBVGGNQF3N4MTUIATJUWK4QZN7SEOWI6SIAW6N7JZOB5G6H33NEQ, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.852728624 (151h11m38.548577377s) [noise=true , address=145.79.227.9:4366 , piece_id=7ZRECKBQC4S6QLERBZY442WYS4H35XWOZYSGGPHSU7NQ6WL4UNVA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.853978774 (151h11m38.549827528s) [noise=true , address=77.90.31.18:4805 , piece_id=U2U34KK4AJK4A5HGNTHEYCIARLAR4X5NLOVCF63TN3PJUIRMMIHA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.854447296 (151h11m38.550296059s) [noise=true , address=77.90.31.16:4734 , piece_id=HFWVFM46PZFLZT6YNZVIA6HKLVN2PTV2GETZEKX356DMDGQO4HOA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.856523089 (151h11m38.552371842s) [noise=true , address=45.91.116.11:4117 , piece_id=QUGUYHEJWEEV733CBC3ONMV7XWDKVH2OFLQWXPQHPKZ7GTH4RLUA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.859490379 (151h11m38.555339122s) [noise=true , address=77.90.31.13:4628 , piece_id=QFTEHZ5FY7A5DPZZ44TRXGFRHPRNJZJHLQZDGRGOWYFEZYIZT2NQ, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.861350291 (151h11m38.557199034s) [noise=true , address=145.79.227.31:4797 , piece_id=2TFSZNURIAU573QUA5DUS7AJZNY2OXC6TJL3ST4VXIWY2WVB5YJA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.863368665 (151h11m38.559217418s) [noise=true , address=145.79.227.23:4547 , piece_id=K54H437KW5AAYUWOTMDBDP5GJ3ZABFKLAAOSS6WS5G5CROYEHODQ, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.864848794 (151h11m38.560697547s) [noise=true , address=145.79.227.24:4581 , piece_id=TSCVCH2KJEVMD4AVOEIHGTMRCRZ6NZMZA2CTQ6FUO2EIL3JAVYEA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.865644930 (151h11m38.561493683s) [noise=true , address=145.79.227.8:4330 , piece_id=4A47HIUX2MTJ2673XZZCMMF4T235TS4JWMNYJEVHHCKAHQX6APMA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:08.560504137 (151h11m40.25635289s) [upload=133242]: Next returned nil segment Nov 5 18:52:08.560522693 (151h11m40.256371446s) [upload=133242]: Waiting for error group managing segments... Nov 5 18:52:08.850333179 (151h11m40.546181932s) [noise=true , address=145.79.227.17:4972 , piece_id=H7UHQV7RDLCZUHY2XMXUAD2PX4ZBLWKTM47N6PH7UOGBRXPMWVOA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.853517822 (151h11m40.549366575s) [noise=true , address=145.79.227.22:4509 , piece_id=7RA5NJ72PI3DT4IC2ECVLZ3A2HXXL546ELVDXHZW4OXNN4D36OEA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.855547508 (151h11m40.551396261s) [noise=true , address=145.79.227.9:4366 , piece_id=7ZRECKBQC4S6QLERBZY442WYS4H35XWOZYSGGPHSU7NQ6WL4UNVA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.856103106 (151h11m40.551951859s) [noise=true , address=145.79.227.25:4618 , piece_id=XRUPY5BUPG7BXN2MI6MGYO4N3GW6TZDJE46MZFNRELWANHHG3N5A, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.860417321 (151h11m40.556266064s) [noise=true , address=145.79.227.8:4330 , piece_id=4A47HIUX2MTJ2673XZZCMMF4T235TS4JWMNYJEVHHCKAHQX6APMA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.865602796 (151h11m40.561451549s) [noise=true , address=145.79.227.23:4547 , piece_id=K54H437KW5AAYUWOTMDBDP5GJ3ZABFKLAAOSS6WS5G5CROYEHODQ, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.879393757 (151h11m40.57524251s) [noise=true , address=145.79.227.31:4797 , piece_id=2TFSZNURIAU573QUA5DUS7AJZNY2OXC6TJL3ST4VXIWY2WVB5YJA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.905541449 (151h11m40.601390202s) [noise=true , address=145.79.227.26:4654 , piece_id=E5J22KNPQSETSY443QQZXFFKUUJFHWFV22SQAVZ4BFQ4M6YWCNQQ, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.929018836 (151h11m40.624867589s) [noise=true , address=145.79.227.24:4581 , piece_id=TSCVCH2KJEVMD4AVOEIHGTMRCRZ6NZMZA2CTQ6FUO2EIL3JAVYEA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:35.899554917 (151h12m7.59540368s) [noise=true , address=77.90.31.17:4771 , piece_id=KVFE7VYW3RZJ25BWSH4BIRRQCMWDYOIK7EJ62XVREGN26YN7XYMQ, seg_pos={0 0}, upload=133242]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: write tcp 188.166.225.170:60676->77.90.31.17:4771: use of closed network connection; write tcp 188.166.225.170:60676->77.90.31.17:4771: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60676->77.90.31.17:4771: use of closed network connection Nov 5 18:52:35.899727506 (151h12m7.595576259s) [noise=true , address=77.90.31.13:4628 , piece_id=QFTEHZ5FY7A5DPZZ44TRXGFRHPRNJZJHLQZDGRGOWYFEZYIZT2NQ, seg_pos={0 0}, upload=133242]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: write tcp 188.166.225.170:42026->77.90.31.13:4628: use of closed network connection; write tcp 188.166.225.170:42026->77.90.31.13:4628: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42026->77.90.31.13:4628: use of closed network connection Nov 5 18:52:35.899860439 (151h12m7.595709192s) [noise=true , address=45.91.116.7:4010 , piece_id=EBVGGNQF3N4MTUIATJUWK4QZN7SEOWI6SIAW6N7JZOB5G6H33NEQ, seg_pos={0 0}, upload=133242]: Done uploading piece. err: ecclient: upload failed (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, address:45.91.116.7:4010): protocol: write tcp 188.166.225.170:40036->45.91.116.7:4010: use of closed network connection; write tcp 188.166.225.170:40036->45.91.116.7:4010: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40036->45.91.116.7:4010: use of closed network connection Nov 5 18:52:35.899923829 (151h12m7.595772572s) [noise=true , address=77.90.31.18:4805 , piece_id=U2U34KK4AJK4A5HGNTHEYCIARLAR4X5NLOVCF63TN3PJUIRMMIHA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: write tcp 188.166.225.170:60758->77.90.31.18:4805: use of closed network connection; write tcp 188.166.225.170:60758->77.90.31.18:4805: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60758->77.90.31.18:4805: use of closed network connection Nov 5 18:52:35.900052084 (151h12m7.595900837s) [noise=true , address=77.90.31.16:4734 , piece_id=HFWVFM46PZFLZT6YNZVIA6HKLVN2PTV2GETZEKX356DMDGQO4HOA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: write tcp 188.166.225.170:50728->77.90.31.16:4734: use of closed network connection; write tcp 188.166.225.170:50728->77.90.31.16:4734: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50728->77.90.31.16:4734: use of closed network connection Nov 5 18:52:35.900231175 (151h12m7.596079938s) [noise=true , address=45.91.116.11:4117 , piece_id=QUGUYHEJWEEV733CBC3ONMV7XWDKVH2OFLQWXPQHPKZ7GTH4RLUA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: ecclient: upload failed (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, address:45.91.116.11:4117): protocol: write tcp 188.166.225.170:41292->45.91.116.11:4117: use of closed network connection; write tcp 188.166.225.170:41292->45.91.116.11:4117: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41292->45.91.116.11:4117: use of closed network connection Nov 5 18:52:35.900342116 (151h12m7.596190859s) [noise=true , address=77.90.31.15:4698 , piece_id=7VACL26N7ZCBXIA2I3XL5Q3UJ2BAHDIPXX4VZCQX3S63ABNW625Q, seg_pos={0 0}, upload=133242]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: write tcp 188.166.225.170:41118->77.90.31.15:4698: use of closed network connection; write tcp 188.166.225.170:41118->77.90.31.15:4698: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41118->77.90.31.15:4698: use of closed network connection Nov 5 18:52:35.900382814 (151h12m7.596231567s) [seg_pos={0 0}, upload=133242]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Nov 5 18:52:36.162442033 (151h12m7.858290786s) [upload=133242]: Done uploading segments... Nov 5 18:52:36.162481568 (151h12m7.858330321s) [upload=133242]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 5.923182883s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [...............................................X..X.........] duration: 5.732769096s trace: https://jaeger.dev.storj.io/trace/3e2ae8884260b157 FAIL (2025-11-05 18:58:42 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.732801498s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [................................................X...X.......] duration: 6.430502636s trace: https://jaeger.dev.storj.io/trace/74cf1e30f6d49950 FAIL (2025-11-05 18:58:41 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.430529165s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.................................................X...X......] duration: 6.138869318s trace: https://jaeger.dev.storj.io/trace/74cf1e30f6d49950 FAIL (2025-11-05 18:58:12 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.138901379s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [..............................................XX...X........] duration: 3.512103507s trace: https://jaeger.dev.storj.io/trace/204ab2570dc82d9c FAIL (2025-11-05 18:58:02 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.512140596s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...............................................X...X........] duration: 5.904450248s trace: https://jaeger.dev.storj.io/trace/2e576aae48e8f03e FAIL (2025-11-05 18:57:54 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== 5 18:52:06.850167887 (151h11m38.54601663s) [noise=true , address=145.79.227.25:4618 , piece_id=XRUPY5BUPG7BXN2MI6MGYO4N3GW6TZDJE46MZFNRELWANHHG3N5A, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.850747301 (151h11m38.546596054s) [noise=true , address=77.90.31.15:4698 , piece_id=7VACL26N7ZCBXIA2I3XL5Q3UJ2BAHDIPXX4VZCQX3S63ABNW625Q, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.851222605 (151h11m38.547071358s) [noise=true , address=145.79.227.17:4972 , piece_id=H7UHQV7RDLCZUHY2XMXUAD2PX4ZBLWKTM47N6PH7UOGBRXPMWVOA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.851558575 (151h11m38.547407319s) [noise=true , address=145.79.227.22:4509 , piece_id=7RA5NJ72PI3DT4IC2ECVLZ3A2HXXL546ELVDXHZW4OXNN4D36OEA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.851909263 (151h11m38.547758006s) [noise=true , address=145.79.227.26:4654 , piece_id=E5J22KNPQSETSY443QQZXFFKUUJFHWFV22SQAVZ4BFQ4M6YWCNQQ, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.852327540 (151h11m38.548176293s) [noise=true , address=45.91.116.7:4010 , piece_id=EBVGGNQF3N4MTUIATJUWK4QZN7SEOWI6SIAW6N7JZOB5G6H33NEQ, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.852728624 (151h11m38.548577377s) [noise=true , address=145.79.227.9:4366 , piece_id=7ZRECKBQC4S6QLERBZY442WYS4H35XWOZYSGGPHSU7NQ6WL4UNVA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.853978774 (151h11m38.549827528s) [noise=true , address=77.90.31.18:4805 , piece_id=U2U34KK4AJK4A5HGNTHEYCIARLAR4X5NLOVCF63TN3PJUIRMMIHA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.854447296 (151h11m38.550296059s) [noise=true , address=77.90.31.16:4734 , piece_id=HFWVFM46PZFLZT6YNZVIA6HKLVN2PTV2GETZEKX356DMDGQO4HOA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.856523089 (151h11m38.552371842s) [noise=true , address=45.91.116.11:4117 , piece_id=QUGUYHEJWEEV733CBC3ONMV7XWDKVH2OFLQWXPQHPKZ7GTH4RLUA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.859490379 (151h11m38.555339122s) [noise=true , address=77.90.31.13:4628 , piece_id=QFTEHZ5FY7A5DPZZ44TRXGFRHPRNJZJHLQZDGRGOWYFEZYIZT2NQ, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.861350291 (151h11m38.557199034s) [noise=true , address=145.79.227.31:4797 , piece_id=2TFSZNURIAU573QUA5DUS7AJZNY2OXC6TJL3ST4VXIWY2WVB5YJA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.863368665 (151h11m38.559217418s) [noise=true , address=145.79.227.23:4547 , piece_id=K54H437KW5AAYUWOTMDBDP5GJ3ZABFKLAAOSS6WS5G5CROYEHODQ, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.864848794 (151h11m38.560697547s) [noise=true , address=145.79.227.24:4581 , piece_id=TSCVCH2KJEVMD4AVOEIHGTMRCRZ6NZMZA2CTQ6FUO2EIL3JAVYEA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.865644930 (151h11m38.561493683s) [noise=true , address=145.79.227.8:4330 , piece_id=4A47HIUX2MTJ2673XZZCMMF4T235TS4JWMNYJEVHHCKAHQX6APMA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:08.560504137 (151h11m40.25635289s) [upload=133242]: Next returned nil segment Nov 5 18:52:08.560522693 (151h11m40.256371446s) [upload=133242]: Waiting for error group managing segments... Nov 5 18:52:08.850333179 (151h11m40.546181932s) [noise=true , address=145.79.227.17:4972 , piece_id=H7UHQV7RDLCZUHY2XMXUAD2PX4ZBLWKTM47N6PH7UOGBRXPMWVOA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.853517822 (151h11m40.549366575s) [noise=true , address=145.79.227.22:4509 , piece_id=7RA5NJ72PI3DT4IC2ECVLZ3A2HXXL546ELVDXHZW4OXNN4D36OEA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.855547508 (151h11m40.551396261s) [noise=true , address=145.79.227.9:4366 , piece_id=7ZRECKBQC4S6QLERBZY442WYS4H35XWOZYSGGPHSU7NQ6WL4UNVA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.856103106 (151h11m40.551951859s) [noise=true , address=145.79.227.25:4618 , piece_id=XRUPY5BUPG7BXN2MI6MGYO4N3GW6TZDJE46MZFNRELWANHHG3N5A, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.860417321 (151h11m40.556266064s) [noise=true , address=145.79.227.8:4330 , piece_id=4A47HIUX2MTJ2673XZZCMMF4T235TS4JWMNYJEVHHCKAHQX6APMA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.865602796 (151h11m40.561451549s) [noise=true , address=145.79.227.23:4547 , piece_id=K54H437KW5AAYUWOTMDBDP5GJ3ZABFKLAAOSS6WS5G5CROYEHODQ, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.879393757 (151h11m40.57524251s) [noise=true , address=145.79.227.31:4797 , piece_id=2TFSZNURIAU573QUA5DUS7AJZNY2OXC6TJL3ST4VXIWY2WVB5YJA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.905541449 (151h11m40.601390202s) [noise=true , address=145.79.227.26:4654 , piece_id=E5J22KNPQSETSY443QQZXFFKUUJFHWFV22SQAVZ4BFQ4M6YWCNQQ, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.929018836 (151h11m40.624867589s) [noise=true , address=145.79.227.24:4581 , piece_id=TSCVCH2KJEVMD4AVOEIHGTMRCRZ6NZMZA2CTQ6FUO2EIL3JAVYEA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:35.899554917 (151h12m7.59540368s) [noise=true , address=77.90.31.17:4771 , piece_id=KVFE7VYW3RZJ25BWSH4BIRRQCMWDYOIK7EJ62XVREGN26YN7XYMQ, seg_pos={0 0}, upload=133242]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: write tcp 188.166.225.170:60676->77.90.31.17:4771: use of closed network connection; write tcp 188.166.225.170:60676->77.90.31.17:4771: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60676->77.90.31.17:4771: use of closed network connection Nov 5 18:52:35.899727506 (151h12m7.595576259s) [noise=true , address=77.90.31.13:4628 , piece_id=QFTEHZ5FY7A5DPZZ44TRXGFRHPRNJZJHLQZDGRGOWYFEZYIZT2NQ, seg_pos={0 0}, upload=133242]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: write tcp 188.166.225.170:42026->77.90.31.13:4628: use of closed network connection; write tcp 188.166.225.170:42026->77.90.31.13:4628: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42026->77.90.31.13:4628: use of closed network connection Nov 5 18:52:35.899860439 (151h12m7.595709192s) [noise=true , address=45.91.116.7:4010 , piece_id=EBVGGNQF3N4MTUIATJUWK4QZN7SEOWI6SIAW6N7JZOB5G6H33NEQ, seg_pos={0 0}, upload=133242]: Done uploading piece. err: ecclient: upload failed (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, address:45.91.116.7:4010): protocol: write tcp 188.166.225.170:40036->45.91.116.7:4010: use of closed network connection; write tcp 188.166.225.170:40036->45.91.116.7:4010: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40036->45.91.116.7:4010: use of closed network connection Nov 5 18:52:35.899923829 (151h12m7.595772572s) [noise=true , address=77.90.31.18:4805 , piece_id=U2U34KK4AJK4A5HGNTHEYCIARLAR4X5NLOVCF63TN3PJUIRMMIHA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: write tcp 188.166.225.170:60758->77.90.31.18:4805: use of closed network connection; write tcp 188.166.225.170:60758->77.90.31.18:4805: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60758->77.90.31.18:4805: use of closed network connection Nov 5 18:52:35.900052084 (151h12m7.595900837s) [noise=true , address=77.90.31.16:4734 , piece_id=HFWVFM46PZFLZT6YNZVIA6HKLVN2PTV2GETZEKX356DMDGQO4HOA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: write tcp 188.166.225.170:50728->77.90.31.16:4734: use of closed network connection; write tcp 188.166.225.170:50728->77.90.31.16:4734: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50728->77.90.31.16:4734: use of closed network connection Nov 5 18:52:35.900231175 (151h12m7.596079938s) [noise=true , address=45.91.116.11:4117 , piece_id=QUGUYHEJWEEV733CBC3ONMV7XWDKVH2OFLQWXPQHPKZ7GTH4RLUA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: ecclient: upload failed (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, address:45.91.116.11:4117): protocol: write tcp 188.166.225.170:41292->45.91.116.11:4117: use of closed network connection; write tcp 188.166.225.170:41292->45.91.116.11:4117: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41292->45.91.116.11:4117: use of closed network connection Nov 5 18:52:35.900342116 (151h12m7.596190859s) [noise=true , address=77.90.31.15:4698 , piece_id=7VACL26N7ZCBXIA2I3XL5Q3UJ2BAHDIPXX4VZCQX3S63ABNW625Q, seg_pos={0 0}, upload=133242]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: write tcp 188.166.225.170:41118->77.90.31.15:4698: use of closed network connection; write tcp 188.166.225.170:41118->77.90.31.15:4698: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41118->77.90.31.15:4698: use of closed network connection Nov 5 18:52:35.900382814 (151h12m7.596231567s) [seg_pos={0 0}, upload=133242]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Nov 5 18:52:36.162442033 (151h12m7.858290786s) [upload=133242]: Done uploading segments... Nov 5 18:52:36.162481568 (151h12m7.858330321s) [upload=133242]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 5.904478532s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [...............................................XX...X.......] duration: 3.553633951s trace: https://jaeger.dev.storj.io/trace/204ab2570dc82d9c FAIL (2025-11-05 18:57:44 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.553688494s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [................................................X..X........] duration: 5.768822021s trace: https://jaeger.dev.storj.io/trace/3e2ae8884260b157 FAIL (2025-11-05 18:57:38 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.768851918s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..................................................X...X.....] duration: 6.226205458s trace: https://jaeger.dev.storj.io/trace/74cf1e30f6d49950 FAIL (2025-11-05 18:57:03 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.226239564s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.................................................X..X.......] duration: 5.657922078s trace: https://jaeger.dev.storj.io/trace/3e2ae8884260b157 FAIL (2025-11-05 18:57:03 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.657943589s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [................................................X...X.......] duration: 5.881608782s trace: https://jaeger.dev.storj.io/trace/2e576aae48e8f03e FAIL (2025-11-05 18:56:56 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== 5 18:52:06.850167887 (151h11m38.54601663s) [noise=true , address=145.79.227.25:4618 , piece_id=XRUPY5BUPG7BXN2MI6MGYO4N3GW6TZDJE46MZFNRELWANHHG3N5A, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.850747301 (151h11m38.546596054s) [noise=true , address=77.90.31.15:4698 , piece_id=7VACL26N7ZCBXIA2I3XL5Q3UJ2BAHDIPXX4VZCQX3S63ABNW625Q, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.851222605 (151h11m38.547071358s) [noise=true , address=145.79.227.17:4972 , piece_id=H7UHQV7RDLCZUHY2XMXUAD2PX4ZBLWKTM47N6PH7UOGBRXPMWVOA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.851558575 (151h11m38.547407319s) [noise=true , address=145.79.227.22:4509 , piece_id=7RA5NJ72PI3DT4IC2ECVLZ3A2HXXL546ELVDXHZW4OXNN4D36OEA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.851909263 (151h11m38.547758006s) [noise=true , address=145.79.227.26:4654 , piece_id=E5J22KNPQSETSY443QQZXFFKUUJFHWFV22SQAVZ4BFQ4M6YWCNQQ, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.852327540 (151h11m38.548176293s) [noise=true , address=45.91.116.7:4010 , piece_id=EBVGGNQF3N4MTUIATJUWK4QZN7SEOWI6SIAW6N7JZOB5G6H33NEQ, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.852728624 (151h11m38.548577377s) [noise=true , address=145.79.227.9:4366 , piece_id=7ZRECKBQC4S6QLERBZY442WYS4H35XWOZYSGGPHSU7NQ6WL4UNVA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.853978774 (151h11m38.549827528s) [noise=true , address=77.90.31.18:4805 , piece_id=U2U34KK4AJK4A5HGNTHEYCIARLAR4X5NLOVCF63TN3PJUIRMMIHA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.854447296 (151h11m38.550296059s) [noise=true , address=77.90.31.16:4734 , piece_id=HFWVFM46PZFLZT6YNZVIA6HKLVN2PTV2GETZEKX356DMDGQO4HOA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.856523089 (151h11m38.552371842s) [noise=true , address=45.91.116.11:4117 , piece_id=QUGUYHEJWEEV733CBC3ONMV7XWDKVH2OFLQWXPQHPKZ7GTH4RLUA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.859490379 (151h11m38.555339122s) [noise=true , address=77.90.31.13:4628 , piece_id=QFTEHZ5FY7A5DPZZ44TRXGFRHPRNJZJHLQZDGRGOWYFEZYIZT2NQ, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.861350291 (151h11m38.557199034s) [noise=true , address=145.79.227.31:4797 , piece_id=2TFSZNURIAU573QUA5DUS7AJZNY2OXC6TJL3ST4VXIWY2WVB5YJA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.863368665 (151h11m38.559217418s) [noise=true , address=145.79.227.23:4547 , piece_id=K54H437KW5AAYUWOTMDBDP5GJ3ZABFKLAAOSS6WS5G5CROYEHODQ, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.864848794 (151h11m38.560697547s) [noise=true , address=145.79.227.24:4581 , piece_id=TSCVCH2KJEVMD4AVOEIHGTMRCRZ6NZMZA2CTQ6FUO2EIL3JAVYEA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.865644930 (151h11m38.561493683s) [noise=true , address=145.79.227.8:4330 , piece_id=4A47HIUX2MTJ2673XZZCMMF4T235TS4JWMNYJEVHHCKAHQX6APMA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:08.560504137 (151h11m40.25635289s) [upload=133242]: Next returned nil segment Nov 5 18:52:08.560522693 (151h11m40.256371446s) [upload=133242]: Waiting for error group managing segments... Nov 5 18:52:08.850333179 (151h11m40.546181932s) [noise=true , address=145.79.227.17:4972 , piece_id=H7UHQV7RDLCZUHY2XMXUAD2PX4ZBLWKTM47N6PH7UOGBRXPMWVOA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.853517822 (151h11m40.549366575s) [noise=true , address=145.79.227.22:4509 , piece_id=7RA5NJ72PI3DT4IC2ECVLZ3A2HXXL546ELVDXHZW4OXNN4D36OEA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.855547508 (151h11m40.551396261s) [noise=true , address=145.79.227.9:4366 , piece_id=7ZRECKBQC4S6QLERBZY442WYS4H35XWOZYSGGPHSU7NQ6WL4UNVA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.856103106 (151h11m40.551951859s) [noise=true , address=145.79.227.25:4618 , piece_id=XRUPY5BUPG7BXN2MI6MGYO4N3GW6TZDJE46MZFNRELWANHHG3N5A, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.860417321 (151h11m40.556266064s) [noise=true , address=145.79.227.8:4330 , piece_id=4A47HIUX2MTJ2673XZZCMMF4T235TS4JWMNYJEVHHCKAHQX6APMA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.865602796 (151h11m40.561451549s) [noise=true , address=145.79.227.23:4547 , piece_id=K54H437KW5AAYUWOTMDBDP5GJ3ZABFKLAAOSS6WS5G5CROYEHODQ, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.879393757 (151h11m40.57524251s) [noise=true , address=145.79.227.31:4797 , piece_id=2TFSZNURIAU573QUA5DUS7AJZNY2OXC6TJL3ST4VXIWY2WVB5YJA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.905541449 (151h11m40.601390202s) [noise=true , address=145.79.227.26:4654 , piece_id=E5J22KNPQSETSY443QQZXFFKUUJFHWFV22SQAVZ4BFQ4M6YWCNQQ, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.929018836 (151h11m40.624867589s) [noise=true , address=145.79.227.24:4581 , piece_id=TSCVCH2KJEVMD4AVOEIHGTMRCRZ6NZMZA2CTQ6FUO2EIL3JAVYEA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:35.899554917 (151h12m7.59540368s) [noise=true , address=77.90.31.17:4771 , piece_id=KVFE7VYW3RZJ25BWSH4BIRRQCMWDYOIK7EJ62XVREGN26YN7XYMQ, seg_pos={0 0}, upload=133242]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: write tcp 188.166.225.170:60676->77.90.31.17:4771: use of closed network connection; write tcp 188.166.225.170:60676->77.90.31.17:4771: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60676->77.90.31.17:4771: use of closed network connection Nov 5 18:52:35.899727506 (151h12m7.595576259s) [noise=true , address=77.90.31.13:4628 , piece_id=QFTEHZ5FY7A5DPZZ44TRXGFRHPRNJZJHLQZDGRGOWYFEZYIZT2NQ, seg_pos={0 0}, upload=133242]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: write tcp 188.166.225.170:42026->77.90.31.13:4628: use of closed network connection; write tcp 188.166.225.170:42026->77.90.31.13:4628: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42026->77.90.31.13:4628: use of closed network connection Nov 5 18:52:35.899860439 (151h12m7.595709192s) [noise=true , address=45.91.116.7:4010 , piece_id=EBVGGNQF3N4MTUIATJUWK4QZN7SEOWI6SIAW6N7JZOB5G6H33NEQ, seg_pos={0 0}, upload=133242]: Done uploading piece. err: ecclient: upload failed (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, address:45.91.116.7:4010): protocol: write tcp 188.166.225.170:40036->45.91.116.7:4010: use of closed network connection; write tcp 188.166.225.170:40036->45.91.116.7:4010: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40036->45.91.116.7:4010: use of closed network connection Nov 5 18:52:35.899923829 (151h12m7.595772572s) [noise=true , address=77.90.31.18:4805 , piece_id=U2U34KK4AJK4A5HGNTHEYCIARLAR4X5NLOVCF63TN3PJUIRMMIHA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: write tcp 188.166.225.170:60758->77.90.31.18:4805: use of closed network connection; write tcp 188.166.225.170:60758->77.90.31.18:4805: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60758->77.90.31.18:4805: use of closed network connection Nov 5 18:52:35.900052084 (151h12m7.595900837s) [noise=true , address=77.90.31.16:4734 , piece_id=HFWVFM46PZFLZT6YNZVIA6HKLVN2PTV2GETZEKX356DMDGQO4HOA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: write tcp 188.166.225.170:50728->77.90.31.16:4734: use of closed network connection; write tcp 188.166.225.170:50728->77.90.31.16:4734: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50728->77.90.31.16:4734: use of closed network connection Nov 5 18:52:35.900231175 (151h12m7.596079938s) [noise=true , address=45.91.116.11:4117 , piece_id=QUGUYHEJWEEV733CBC3ONMV7XWDKVH2OFLQWXPQHPKZ7GTH4RLUA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: ecclient: upload failed (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, address:45.91.116.11:4117): protocol: write tcp 188.166.225.170:41292->45.91.116.11:4117: use of closed network connection; write tcp 188.166.225.170:41292->45.91.116.11:4117: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41292->45.91.116.11:4117: use of closed network connection Nov 5 18:52:35.900342116 (151h12m7.596190859s) [noise=true , address=77.90.31.15:4698 , piece_id=7VACL26N7ZCBXIA2I3XL5Q3UJ2BAHDIPXX4VZCQX3S63ABNW625Q, seg_pos={0 0}, upload=133242]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: write tcp 188.166.225.170:41118->77.90.31.15:4698: use of closed network connection; write tcp 188.166.225.170:41118->77.90.31.15:4698: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41118->77.90.31.15:4698: use of closed network connection Nov 5 18:52:35.900382814 (151h12m7.596231567s) [seg_pos={0 0}, upload=133242]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Nov 5 18:52:36.162442033 (151h12m7.858290786s) [upload=133242]: Done uploading segments... Nov 5 18:52:36.162481568 (151h12m7.858330321s) [upload=133242]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 5.881640172s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [..................................................X..X......] duration: 5.851378963s trace: https://jaeger.dev.storj.io/trace/3e2ae8884260b157 FAIL (2025-11-05 18:56:42 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.851406245s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [................................................XX...X......] duration: 3.424729244s trace: https://jaeger.dev.storj.io/trace/204ab2570dc82d9c FAIL (2025-11-05 18:56:37 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.424756285s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [.................................................XX...X.....] duration: 3.451124577s trace: https://jaeger.dev.storj.io/trace/204ab2570dc82d9c FAIL (2025-11-05 18:56:10 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.451160223s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...................................................X...X....] duration: 6.360344743s trace: https://jaeger.dev.storj.io/trace/74cf1e30f6d49950 FAIL (2025-11-05 18:55:56 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.360373676s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.................................................X...X......] duration: 5.951129112s trace: https://jaeger.dev.storj.io/trace/2e576aae48e8f03e FAIL (2025-11-05 18:55:48 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== 5 18:52:06.850167887 (151h11m38.54601663s) [noise=true , address=145.79.227.25:4618 , piece_id=XRUPY5BUPG7BXN2MI6MGYO4N3GW6TZDJE46MZFNRELWANHHG3N5A, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.850747301 (151h11m38.546596054s) [noise=true , address=77.90.31.15:4698 , piece_id=7VACL26N7ZCBXIA2I3XL5Q3UJ2BAHDIPXX4VZCQX3S63ABNW625Q, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.851222605 (151h11m38.547071358s) [noise=true , address=145.79.227.17:4972 , piece_id=H7UHQV7RDLCZUHY2XMXUAD2PX4ZBLWKTM47N6PH7UOGBRXPMWVOA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.851558575 (151h11m38.547407319s) [noise=true , address=145.79.227.22:4509 , piece_id=7RA5NJ72PI3DT4IC2ECVLZ3A2HXXL546ELVDXHZW4OXNN4D36OEA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.851909263 (151h11m38.547758006s) [noise=true , address=145.79.227.26:4654 , piece_id=E5J22KNPQSETSY443QQZXFFKUUJFHWFV22SQAVZ4BFQ4M6YWCNQQ, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.852327540 (151h11m38.548176293s) [noise=true , address=45.91.116.7:4010 , piece_id=EBVGGNQF3N4MTUIATJUWK4QZN7SEOWI6SIAW6N7JZOB5G6H33NEQ, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.852728624 (151h11m38.548577377s) [noise=true , address=145.79.227.9:4366 , piece_id=7ZRECKBQC4S6QLERBZY442WYS4H35XWOZYSGGPHSU7NQ6WL4UNVA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.853978774 (151h11m38.549827528s) [noise=true , address=77.90.31.18:4805 , piece_id=U2U34KK4AJK4A5HGNTHEYCIARLAR4X5NLOVCF63TN3PJUIRMMIHA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.854447296 (151h11m38.550296059s) [noise=true , address=77.90.31.16:4734 , piece_id=HFWVFM46PZFLZT6YNZVIA6HKLVN2PTV2GETZEKX356DMDGQO4HOA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.856523089 (151h11m38.552371842s) [noise=true , address=45.91.116.11:4117 , piece_id=QUGUYHEJWEEV733CBC3ONMV7XWDKVH2OFLQWXPQHPKZ7GTH4RLUA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.859490379 (151h11m38.555339122s) [noise=true , address=77.90.31.13:4628 , piece_id=QFTEHZ5FY7A5DPZZ44TRXGFRHPRNJZJHLQZDGRGOWYFEZYIZT2NQ, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.861350291 (151h11m38.557199034s) [noise=true , address=145.79.227.31:4797 , piece_id=2TFSZNURIAU573QUA5DUS7AJZNY2OXC6TJL3ST4VXIWY2WVB5YJA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.863368665 (151h11m38.559217418s) [noise=true , address=145.79.227.23:4547 , piece_id=K54H437KW5AAYUWOTMDBDP5GJ3ZABFKLAAOSS6WS5G5CROYEHODQ, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.864848794 (151h11m38.560697547s) [noise=true , address=145.79.227.24:4581 , piece_id=TSCVCH2KJEVMD4AVOEIHGTMRCRZ6NZMZA2CTQ6FUO2EIL3JAVYEA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.865644930 (151h11m38.561493683s) [noise=true , address=145.79.227.8:4330 , piece_id=4A47HIUX2MTJ2673XZZCMMF4T235TS4JWMNYJEVHHCKAHQX6APMA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:08.560504137 (151h11m40.25635289s) [upload=133242]: Next returned nil segment Nov 5 18:52:08.560522693 (151h11m40.256371446s) [upload=133242]: Waiting for error group managing segments... Nov 5 18:52:08.850333179 (151h11m40.546181932s) [noise=true , address=145.79.227.17:4972 , piece_id=H7UHQV7RDLCZUHY2XMXUAD2PX4ZBLWKTM47N6PH7UOGBRXPMWVOA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.853517822 (151h11m40.549366575s) [noise=true , address=145.79.227.22:4509 , piece_id=7RA5NJ72PI3DT4IC2ECVLZ3A2HXXL546ELVDXHZW4OXNN4D36OEA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.855547508 (151h11m40.551396261s) [noise=true , address=145.79.227.9:4366 , piece_id=7ZRECKBQC4S6QLERBZY442WYS4H35XWOZYSGGPHSU7NQ6WL4UNVA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.856103106 (151h11m40.551951859s) [noise=true , address=145.79.227.25:4618 , piece_id=XRUPY5BUPG7BXN2MI6MGYO4N3GW6TZDJE46MZFNRELWANHHG3N5A, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.860417321 (151h11m40.556266064s) [noise=true , address=145.79.227.8:4330 , piece_id=4A47HIUX2MTJ2673XZZCMMF4T235TS4JWMNYJEVHHCKAHQX6APMA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.865602796 (151h11m40.561451549s) [noise=true , address=145.79.227.23:4547 , piece_id=K54H437KW5AAYUWOTMDBDP5GJ3ZABFKLAAOSS6WS5G5CROYEHODQ, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.879393757 (151h11m40.57524251s) [noise=true , address=145.79.227.31:4797 , piece_id=2TFSZNURIAU573QUA5DUS7AJZNY2OXC6TJL3ST4VXIWY2WVB5YJA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.905541449 (151h11m40.601390202s) [noise=true , address=145.79.227.26:4654 , piece_id=E5J22KNPQSETSY443QQZXFFKUUJFHWFV22SQAVZ4BFQ4M6YWCNQQ, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.929018836 (151h11m40.624867589s) [noise=true , address=145.79.227.24:4581 , piece_id=TSCVCH2KJEVMD4AVOEIHGTMRCRZ6NZMZA2CTQ6FUO2EIL3JAVYEA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:35.899554917 (151h12m7.59540368s) [noise=true , address=77.90.31.17:4771 , piece_id=KVFE7VYW3RZJ25BWSH4BIRRQCMWDYOIK7EJ62XVREGN26YN7XYMQ, seg_pos={0 0}, upload=133242]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: write tcp 188.166.225.170:60676->77.90.31.17:4771: use of closed network connection; write tcp 188.166.225.170:60676->77.90.31.17:4771: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60676->77.90.31.17:4771: use of closed network connection Nov 5 18:52:35.899727506 (151h12m7.595576259s) [noise=true , address=77.90.31.13:4628 , piece_id=QFTEHZ5FY7A5DPZZ44TRXGFRHPRNJZJHLQZDGRGOWYFEZYIZT2NQ, seg_pos={0 0}, upload=133242]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: write tcp 188.166.225.170:42026->77.90.31.13:4628: use of closed network connection; write tcp 188.166.225.170:42026->77.90.31.13:4628: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42026->77.90.31.13:4628: use of closed network connection Nov 5 18:52:35.899860439 (151h12m7.595709192s) [noise=true , address=45.91.116.7:4010 , piece_id=EBVGGNQF3N4MTUIATJUWK4QZN7SEOWI6SIAW6N7JZOB5G6H33NEQ, seg_pos={0 0}, upload=133242]: Done uploading piece. err: ecclient: upload failed (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, address:45.91.116.7:4010): protocol: write tcp 188.166.225.170:40036->45.91.116.7:4010: use of closed network connection; write tcp 188.166.225.170:40036->45.91.116.7:4010: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40036->45.91.116.7:4010: use of closed network connection Nov 5 18:52:35.899923829 (151h12m7.595772572s) [noise=true , address=77.90.31.18:4805 , piece_id=U2U34KK4AJK4A5HGNTHEYCIARLAR4X5NLOVCF63TN3PJUIRMMIHA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: write tcp 188.166.225.170:60758->77.90.31.18:4805: use of closed network connection; write tcp 188.166.225.170:60758->77.90.31.18:4805: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60758->77.90.31.18:4805: use of closed network connection Nov 5 18:52:35.900052084 (151h12m7.595900837s) [noise=true , address=77.90.31.16:4734 , piece_id=HFWVFM46PZFLZT6YNZVIA6HKLVN2PTV2GETZEKX356DMDGQO4HOA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: write tcp 188.166.225.170:50728->77.90.31.16:4734: use of closed network connection; write tcp 188.166.225.170:50728->77.90.31.16:4734: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50728->77.90.31.16:4734: use of closed network connection Nov 5 18:52:35.900231175 (151h12m7.596079938s) [noise=true , address=45.91.116.11:4117 , piece_id=QUGUYHEJWEEV733CBC3ONMV7XWDKVH2OFLQWXPQHPKZ7GTH4RLUA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: ecclient: upload failed (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, address:45.91.116.11:4117): protocol: write tcp 188.166.225.170:41292->45.91.116.11:4117: use of closed network connection; write tcp 188.166.225.170:41292->45.91.116.11:4117: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41292->45.91.116.11:4117: use of closed network connection Nov 5 18:52:35.900342116 (151h12m7.596190859s) [noise=true , address=77.90.31.15:4698 , piece_id=7VACL26N7ZCBXIA2I3XL5Q3UJ2BAHDIPXX4VZCQX3S63ABNW625Q, seg_pos={0 0}, upload=133242]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: write tcp 188.166.225.170:41118->77.90.31.15:4698: use of closed network connection; write tcp 188.166.225.170:41118->77.90.31.15:4698: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41118->77.90.31.15:4698: use of closed network connection Nov 5 18:52:35.900382814 (151h12m7.596231567s) [seg_pos={0 0}, upload=133242]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Nov 5 18:52:36.162442033 (151h12m7.858290786s) [upload=133242]: Done uploading segments... Nov 5 18:52:36.162481568 (151h12m7.858330321s) [upload=133242]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 5.951162766s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [...................................................X..X.....] duration: 5.85248543s trace: https://jaeger.dev.storj.io/trace/3e2ae8884260b157 FAIL (2025-11-05 18:55:34 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.852578065s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [..................................................XX...X....] duration: 3.441179298s trace: https://jaeger.dev.storj.io/trace/204ab2570dc82d9c FAIL (2025-11-05 18:55:24 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.441213512s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [...................................................XX...X...] duration: 3.60110892s trace: https://jaeger.dev.storj.io/trace/204ab2570dc82d9c FAIL (2025-11-05 18:55:07 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.60158119s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [....................................................X..X....] duration: 6.432613178s trace: https://jaeger.dev.storj.io/trace/3e2ae8884260b157 FAIL (2025-11-05 18:55:06 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.432648396s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [....................................................X...X...] duration: 6.253813579s trace: https://jaeger.dev.storj.io/trace/74cf1e30f6d49950 FAIL (2025-11-05 18:54:52 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.2538459s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..................................................X...X.....] duration: 5.836970574s trace: https://jaeger.dev.storj.io/trace/2e576aae48e8f03e FAIL (2025-11-05 18:54:45 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== 5 18:52:06.850167887 (151h11m38.54601663s) [noise=true , address=145.79.227.25:4618 , piece_id=XRUPY5BUPG7BXN2MI6MGYO4N3GW6TZDJE46MZFNRELWANHHG3N5A, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.850747301 (151h11m38.546596054s) [noise=true , address=77.90.31.15:4698 , piece_id=7VACL26N7ZCBXIA2I3XL5Q3UJ2BAHDIPXX4VZCQX3S63ABNW625Q, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.851222605 (151h11m38.547071358s) [noise=true , address=145.79.227.17:4972 , piece_id=H7UHQV7RDLCZUHY2XMXUAD2PX4ZBLWKTM47N6PH7UOGBRXPMWVOA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.851558575 (151h11m38.547407319s) [noise=true , address=145.79.227.22:4509 , piece_id=7RA5NJ72PI3DT4IC2ECVLZ3A2HXXL546ELVDXHZW4OXNN4D36OEA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.851909263 (151h11m38.547758006s) [noise=true , address=145.79.227.26:4654 , piece_id=E5J22KNPQSETSY443QQZXFFKUUJFHWFV22SQAVZ4BFQ4M6YWCNQQ, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.852327540 (151h11m38.548176293s) [noise=true , address=45.91.116.7:4010 , piece_id=EBVGGNQF3N4MTUIATJUWK4QZN7SEOWI6SIAW6N7JZOB5G6H33NEQ, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.852728624 (151h11m38.548577377s) [noise=true , address=145.79.227.9:4366 , piece_id=7ZRECKBQC4S6QLERBZY442WYS4H35XWOZYSGGPHSU7NQ6WL4UNVA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.853978774 (151h11m38.549827528s) [noise=true , address=77.90.31.18:4805 , piece_id=U2U34KK4AJK4A5HGNTHEYCIARLAR4X5NLOVCF63TN3PJUIRMMIHA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.854447296 (151h11m38.550296059s) [noise=true , address=77.90.31.16:4734 , piece_id=HFWVFM46PZFLZT6YNZVIA6HKLVN2PTV2GETZEKX356DMDGQO4HOA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.856523089 (151h11m38.552371842s) [noise=true , address=45.91.116.11:4117 , piece_id=QUGUYHEJWEEV733CBC3ONMV7XWDKVH2OFLQWXPQHPKZ7GTH4RLUA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.859490379 (151h11m38.555339122s) [noise=true , address=77.90.31.13:4628 , piece_id=QFTEHZ5FY7A5DPZZ44TRXGFRHPRNJZJHLQZDGRGOWYFEZYIZT2NQ, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.861350291 (151h11m38.557199034s) [noise=true , address=145.79.227.31:4797 , piece_id=2TFSZNURIAU573QUA5DUS7AJZNY2OXC6TJL3ST4VXIWY2WVB5YJA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.863368665 (151h11m38.559217418s) [noise=true , address=145.79.227.23:4547 , piece_id=K54H437KW5AAYUWOTMDBDP5GJ3ZABFKLAAOSS6WS5G5CROYEHODQ, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.864848794 (151h11m38.560697547s) [noise=true , address=145.79.227.24:4581 , piece_id=TSCVCH2KJEVMD4AVOEIHGTMRCRZ6NZMZA2CTQ6FUO2EIL3JAVYEA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.865644930 (151h11m38.561493683s) [noise=true , address=145.79.227.8:4330 , piece_id=4A47HIUX2MTJ2673XZZCMMF4T235TS4JWMNYJEVHHCKAHQX6APMA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:08.560504137 (151h11m40.25635289s) [upload=133242]: Next returned nil segment Nov 5 18:52:08.560522693 (151h11m40.256371446s) [upload=133242]: Waiting for error group managing segments... Nov 5 18:52:08.850333179 (151h11m40.546181932s) [noise=true , address=145.79.227.17:4972 , piece_id=H7UHQV7RDLCZUHY2XMXUAD2PX4ZBLWKTM47N6PH7UOGBRXPMWVOA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.853517822 (151h11m40.549366575s) [noise=true , address=145.79.227.22:4509 , piece_id=7RA5NJ72PI3DT4IC2ECVLZ3A2HXXL546ELVDXHZW4OXNN4D36OEA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.855547508 (151h11m40.551396261s) [noise=true , address=145.79.227.9:4366 , piece_id=7ZRECKBQC4S6QLERBZY442WYS4H35XWOZYSGGPHSU7NQ6WL4UNVA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.856103106 (151h11m40.551951859s) [noise=true , address=145.79.227.25:4618 , piece_id=XRUPY5BUPG7BXN2MI6MGYO4N3GW6TZDJE46MZFNRELWANHHG3N5A, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.860417321 (151h11m40.556266064s) [noise=true , address=145.79.227.8:4330 , piece_id=4A47HIUX2MTJ2673XZZCMMF4T235TS4JWMNYJEVHHCKAHQX6APMA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.865602796 (151h11m40.561451549s) [noise=true , address=145.79.227.23:4547 , piece_id=K54H437KW5AAYUWOTMDBDP5GJ3ZABFKLAAOSS6WS5G5CROYEHODQ, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.879393757 (151h11m40.57524251s) [noise=true , address=145.79.227.31:4797 , piece_id=2TFSZNURIAU573QUA5DUS7AJZNY2OXC6TJL3ST4VXIWY2WVB5YJA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.905541449 (151h11m40.601390202s) [noise=true , address=145.79.227.26:4654 , piece_id=E5J22KNPQSETSY443QQZXFFKUUJFHWFV22SQAVZ4BFQ4M6YWCNQQ, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.929018836 (151h11m40.624867589s) [noise=true , address=145.79.227.24:4581 , piece_id=TSCVCH2KJEVMD4AVOEIHGTMRCRZ6NZMZA2CTQ6FUO2EIL3JAVYEA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:35.899554917 (151h12m7.59540368s) [noise=true , address=77.90.31.17:4771 , piece_id=KVFE7VYW3RZJ25BWSH4BIRRQCMWDYOIK7EJ62XVREGN26YN7XYMQ, seg_pos={0 0}, upload=133242]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: write tcp 188.166.225.170:60676->77.90.31.17:4771: use of closed network connection; write tcp 188.166.225.170:60676->77.90.31.17:4771: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60676->77.90.31.17:4771: use of closed network connection Nov 5 18:52:35.899727506 (151h12m7.595576259s) [noise=true , address=77.90.31.13:4628 , piece_id=QFTEHZ5FY7A5DPZZ44TRXGFRHPRNJZJHLQZDGRGOWYFEZYIZT2NQ, seg_pos={0 0}, upload=133242]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: write tcp 188.166.225.170:42026->77.90.31.13:4628: use of closed network connection; write tcp 188.166.225.170:42026->77.90.31.13:4628: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42026->77.90.31.13:4628: use of closed network connection Nov 5 18:52:35.899860439 (151h12m7.595709192s) [noise=true , address=45.91.116.7:4010 , piece_id=EBVGGNQF3N4MTUIATJUWK4QZN7SEOWI6SIAW6N7JZOB5G6H33NEQ, seg_pos={0 0}, upload=133242]: Done uploading piece. err: ecclient: upload failed (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, address:45.91.116.7:4010): protocol: write tcp 188.166.225.170:40036->45.91.116.7:4010: use of closed network connection; write tcp 188.166.225.170:40036->45.91.116.7:4010: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40036->45.91.116.7:4010: use of closed network connection Nov 5 18:52:35.899923829 (151h12m7.595772572s) [noise=true , address=77.90.31.18:4805 , piece_id=U2U34KK4AJK4A5HGNTHEYCIARLAR4X5NLOVCF63TN3PJUIRMMIHA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: write tcp 188.166.225.170:60758->77.90.31.18:4805: use of closed network connection; write tcp 188.166.225.170:60758->77.90.31.18:4805: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60758->77.90.31.18:4805: use of closed network connection Nov 5 18:52:35.900052084 (151h12m7.595900837s) [noise=true , address=77.90.31.16:4734 , piece_id=HFWVFM46PZFLZT6YNZVIA6HKLVN2PTV2GETZEKX356DMDGQO4HOA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: write tcp 188.166.225.170:50728->77.90.31.16:4734: use of closed network connection; write tcp 188.166.225.170:50728->77.90.31.16:4734: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50728->77.90.31.16:4734: use of closed network connection Nov 5 18:52:35.900231175 (151h12m7.596079938s) [noise=true , address=45.91.116.11:4117 , piece_id=QUGUYHEJWEEV733CBC3ONMV7XWDKVH2OFLQWXPQHPKZ7GTH4RLUA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: ecclient: upload failed (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, address:45.91.116.11:4117): protocol: write tcp 188.166.225.170:41292->45.91.116.11:4117: use of closed network connection; write tcp 188.166.225.170:41292->45.91.116.11:4117: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41292->45.91.116.11:4117: use of closed network connection Nov 5 18:52:35.900342116 (151h12m7.596190859s) [noise=true , address=77.90.31.15:4698 , piece_id=7VACL26N7ZCBXIA2I3XL5Q3UJ2BAHDIPXX4VZCQX3S63ABNW625Q, seg_pos={0 0}, upload=133242]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: write tcp 188.166.225.170:41118->77.90.31.15:4698: use of closed network connection; write tcp 188.166.225.170:41118->77.90.31.15:4698: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41118->77.90.31.15:4698: use of closed network connection Nov 5 18:52:35.900382814 (151h12m7.596231567s) [seg_pos={0 0}, upload=133242]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Nov 5 18:52:36.162442033 (151h12m7.858290786s) [upload=133242]: Done uploading segments... Nov 5 18:52:36.162481568 (151h12m7.858330321s) [upload=133242]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 5.837000451s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...................................................X...X....] duration: 5.895184646s trace: https://jaeger.dev.storj.io/trace/2e576aae48e8f03e FAIL (2025-11-05 18:54:09 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== 5 18:52:06.850167887 (151h11m38.54601663s) [noise=true , address=145.79.227.25:4618 , piece_id=XRUPY5BUPG7BXN2MI6MGYO4N3GW6TZDJE46MZFNRELWANHHG3N5A, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.850747301 (151h11m38.546596054s) [noise=true , address=77.90.31.15:4698 , piece_id=7VACL26N7ZCBXIA2I3XL5Q3UJ2BAHDIPXX4VZCQX3S63ABNW625Q, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.851222605 (151h11m38.547071358s) [noise=true , address=145.79.227.17:4972 , piece_id=H7UHQV7RDLCZUHY2XMXUAD2PX4ZBLWKTM47N6PH7UOGBRXPMWVOA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.851558575 (151h11m38.547407319s) [noise=true , address=145.79.227.22:4509 , piece_id=7RA5NJ72PI3DT4IC2ECVLZ3A2HXXL546ELVDXHZW4OXNN4D36OEA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.851909263 (151h11m38.547758006s) [noise=true , address=145.79.227.26:4654 , piece_id=E5J22KNPQSETSY443QQZXFFKUUJFHWFV22SQAVZ4BFQ4M6YWCNQQ, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.852327540 (151h11m38.548176293s) [noise=true , address=45.91.116.7:4010 , piece_id=EBVGGNQF3N4MTUIATJUWK4QZN7SEOWI6SIAW6N7JZOB5G6H33NEQ, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.852728624 (151h11m38.548577377s) [noise=true , address=145.79.227.9:4366 , piece_id=7ZRECKBQC4S6QLERBZY442WYS4H35XWOZYSGGPHSU7NQ6WL4UNVA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.853978774 (151h11m38.549827528s) [noise=true , address=77.90.31.18:4805 , piece_id=U2U34KK4AJK4A5HGNTHEYCIARLAR4X5NLOVCF63TN3PJUIRMMIHA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.854447296 (151h11m38.550296059s) [noise=true , address=77.90.31.16:4734 , piece_id=HFWVFM46PZFLZT6YNZVIA6HKLVN2PTV2GETZEKX356DMDGQO4HOA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.856523089 (151h11m38.552371842s) [noise=true , address=45.91.116.11:4117 , piece_id=QUGUYHEJWEEV733CBC3ONMV7XWDKVH2OFLQWXPQHPKZ7GTH4RLUA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.859490379 (151h11m38.555339122s) [noise=true , address=77.90.31.13:4628 , piece_id=QFTEHZ5FY7A5DPZZ44TRXGFRHPRNJZJHLQZDGRGOWYFEZYIZT2NQ, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.861350291 (151h11m38.557199034s) [noise=true , address=145.79.227.31:4797 , piece_id=2TFSZNURIAU573QUA5DUS7AJZNY2OXC6TJL3ST4VXIWY2WVB5YJA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.863368665 (151h11m38.559217418s) [noise=true , address=145.79.227.23:4547 , piece_id=K54H437KW5AAYUWOTMDBDP5GJ3ZABFKLAAOSS6WS5G5CROYEHODQ, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.864848794 (151h11m38.560697547s) [noise=true , address=145.79.227.24:4581 , piece_id=TSCVCH2KJEVMD4AVOEIHGTMRCRZ6NZMZA2CTQ6FUO2EIL3JAVYEA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.865644930 (151h11m38.561493683s) [noise=true , address=145.79.227.8:4330 , piece_id=4A47HIUX2MTJ2673XZZCMMF4T235TS4JWMNYJEVHHCKAHQX6APMA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:08.560504137 (151h11m40.25635289s) [upload=133242]: Next returned nil segment Nov 5 18:52:08.560522693 (151h11m40.256371446s) [upload=133242]: Waiting for error group managing segments... Nov 5 18:52:08.850333179 (151h11m40.546181932s) [noise=true , address=145.79.227.17:4972 , piece_id=H7UHQV7RDLCZUHY2XMXUAD2PX4ZBLWKTM47N6PH7UOGBRXPMWVOA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.853517822 (151h11m40.549366575s) [noise=true , address=145.79.227.22:4509 , piece_id=7RA5NJ72PI3DT4IC2ECVLZ3A2HXXL546ELVDXHZW4OXNN4D36OEA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.855547508 (151h11m40.551396261s) [noise=true , address=145.79.227.9:4366 , piece_id=7ZRECKBQC4S6QLERBZY442WYS4H35XWOZYSGGPHSU7NQ6WL4UNVA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.856103106 (151h11m40.551951859s) [noise=true , address=145.79.227.25:4618 , piece_id=XRUPY5BUPG7BXN2MI6MGYO4N3GW6TZDJE46MZFNRELWANHHG3N5A, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.860417321 (151h11m40.556266064s) [noise=true , address=145.79.227.8:4330 , piece_id=4A47HIUX2MTJ2673XZZCMMF4T235TS4JWMNYJEVHHCKAHQX6APMA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.865602796 (151h11m40.561451549s) [noise=true , address=145.79.227.23:4547 , piece_id=K54H437KW5AAYUWOTMDBDP5GJ3ZABFKLAAOSS6WS5G5CROYEHODQ, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.879393757 (151h11m40.57524251s) [noise=true , address=145.79.227.31:4797 , piece_id=2TFSZNURIAU573QUA5DUS7AJZNY2OXC6TJL3ST4VXIWY2WVB5YJA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.905541449 (151h11m40.601390202s) [noise=true , address=145.79.227.26:4654 , piece_id=E5J22KNPQSETSY443QQZXFFKUUJFHWFV22SQAVZ4BFQ4M6YWCNQQ, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.929018836 (151h11m40.624867589s) [noise=true , address=145.79.227.24:4581 , piece_id=TSCVCH2KJEVMD4AVOEIHGTMRCRZ6NZMZA2CTQ6FUO2EIL3JAVYEA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:35.899554917 (151h12m7.59540368s) [noise=true , address=77.90.31.17:4771 , piece_id=KVFE7VYW3RZJ25BWSH4BIRRQCMWDYOIK7EJ62XVREGN26YN7XYMQ, seg_pos={0 0}, upload=133242]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: write tcp 188.166.225.170:60676->77.90.31.17:4771: use of closed network connection; write tcp 188.166.225.170:60676->77.90.31.17:4771: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60676->77.90.31.17:4771: use of closed network connection Nov 5 18:52:35.899727506 (151h12m7.595576259s) [noise=true , address=77.90.31.13:4628 , piece_id=QFTEHZ5FY7A5DPZZ44TRXGFRHPRNJZJHLQZDGRGOWYFEZYIZT2NQ, seg_pos={0 0}, upload=133242]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: write tcp 188.166.225.170:42026->77.90.31.13:4628: use of closed network connection; write tcp 188.166.225.170:42026->77.90.31.13:4628: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42026->77.90.31.13:4628: use of closed network connection Nov 5 18:52:35.899860439 (151h12m7.595709192s) [noise=true , address=45.91.116.7:4010 , piece_id=EBVGGNQF3N4MTUIATJUWK4QZN7SEOWI6SIAW6N7JZOB5G6H33NEQ, seg_pos={0 0}, upload=133242]: Done uploading piece. err: ecclient: upload failed (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, address:45.91.116.7:4010): protocol: write tcp 188.166.225.170:40036->45.91.116.7:4010: use of closed network connection; write tcp 188.166.225.170:40036->45.91.116.7:4010: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40036->45.91.116.7:4010: use of closed network connection Nov 5 18:52:35.899923829 (151h12m7.595772572s) [noise=true , address=77.90.31.18:4805 , piece_id=U2U34KK4AJK4A5HGNTHEYCIARLAR4X5NLOVCF63TN3PJUIRMMIHA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: write tcp 188.166.225.170:60758->77.90.31.18:4805: use of closed network connection; write tcp 188.166.225.170:60758->77.90.31.18:4805: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60758->77.90.31.18:4805: use of closed network connection Nov 5 18:52:35.900052084 (151h12m7.595900837s) [noise=true , address=77.90.31.16:4734 , piece_id=HFWVFM46PZFLZT6YNZVIA6HKLVN2PTV2GETZEKX356DMDGQO4HOA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: write tcp 188.166.225.170:50728->77.90.31.16:4734: use of closed network connection; write tcp 188.166.225.170:50728->77.90.31.16:4734: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50728->77.90.31.16:4734: use of closed network connection Nov 5 18:52:35.900231175 (151h12m7.596079938s) [noise=true , address=45.91.116.11:4117 , piece_id=QUGUYHEJWEEV733CBC3ONMV7XWDKVH2OFLQWXPQHPKZ7GTH4RLUA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: ecclient: upload failed (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, address:45.91.116.11:4117): protocol: write tcp 188.166.225.170:41292->45.91.116.11:4117: use of closed network connection; write tcp 188.166.225.170:41292->45.91.116.11:4117: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41292->45.91.116.11:4117: use of closed network connection Nov 5 18:52:35.900342116 (151h12m7.596190859s) [noise=true , address=77.90.31.15:4698 , piece_id=7VACL26N7ZCBXIA2I3XL5Q3UJ2BAHDIPXX4VZCQX3S63ABNW625Q, seg_pos={0 0}, upload=133242]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: write tcp 188.166.225.170:41118->77.90.31.15:4698: use of closed network connection; write tcp 188.166.225.170:41118->77.90.31.15:4698: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41118->77.90.31.15:4698: use of closed network connection Nov 5 18:52:35.900382814 (151h12m7.596231567s) [seg_pos={0 0}, upload=133242]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Nov 5 18:52:36.162442033 (151h12m7.858290786s) [upload=133242]: Done uploading segments... Nov 5 18:52:36.162481568 (151h12m7.858330321s) [upload=133242]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 5.895216718s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.....................................................X..X...] duration: 5.915543096s trace: https://jaeger.dev.storj.io/trace/3e2ae8884260b157 FAIL (2025-11-05 18:54:00 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.915569266s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [....................................................XX...X..] duration: 3.475670819s trace: https://jaeger.dev.storj.io/trace/204ab2570dc82d9c FAIL (2025-11-05 18:53:52 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.475690415s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)