Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...X...X....................................................] duration: 5.835895007s trace: https://jaeger.dev.storj.io/trace/2e576aae48e8f03e FAIL (2025-11-05 19:28:13 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== 5 18:52:06.850167887 (151h11m38.54601663s) [noise=true , address=145.79.227.25:4618 , piece_id=XRUPY5BUPG7BXN2MI6MGYO4N3GW6TZDJE46MZFNRELWANHHG3N5A, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.850747301 (151h11m38.546596054s) [noise=true , address=77.90.31.15:4698 , piece_id=7VACL26N7ZCBXIA2I3XL5Q3UJ2BAHDIPXX4VZCQX3S63ABNW625Q, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.851222605 (151h11m38.547071358s) [noise=true , address=145.79.227.17:4972 , piece_id=H7UHQV7RDLCZUHY2XMXUAD2PX4ZBLWKTM47N6PH7UOGBRXPMWVOA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.851558575 (151h11m38.547407319s) [noise=true , address=145.79.227.22:4509 , piece_id=7RA5NJ72PI3DT4IC2ECVLZ3A2HXXL546ELVDXHZW4OXNN4D36OEA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.851909263 (151h11m38.547758006s) [noise=true , address=145.79.227.26:4654 , piece_id=E5J22KNPQSETSY443QQZXFFKUUJFHWFV22SQAVZ4BFQ4M6YWCNQQ, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.852327540 (151h11m38.548176293s) [noise=true , address=45.91.116.7:4010 , piece_id=EBVGGNQF3N4MTUIATJUWK4QZN7SEOWI6SIAW6N7JZOB5G6H33NEQ, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.852728624 (151h11m38.548577377s) [noise=true , address=145.79.227.9:4366 , piece_id=7ZRECKBQC4S6QLERBZY442WYS4H35XWOZYSGGPHSU7NQ6WL4UNVA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.853978774 (151h11m38.549827528s) [noise=true , address=77.90.31.18:4805 , piece_id=U2U34KK4AJK4A5HGNTHEYCIARLAR4X5NLOVCF63TN3PJUIRMMIHA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.854447296 (151h11m38.550296059s) [noise=true , address=77.90.31.16:4734 , piece_id=HFWVFM46PZFLZT6YNZVIA6HKLVN2PTV2GETZEKX356DMDGQO4HOA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.856523089 (151h11m38.552371842s) [noise=true , address=45.91.116.11:4117 , piece_id=QUGUYHEJWEEV733CBC3ONMV7XWDKVH2OFLQWXPQHPKZ7GTH4RLUA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.859490379 (151h11m38.555339122s) [noise=true , address=77.90.31.13:4628 , piece_id=QFTEHZ5FY7A5DPZZ44TRXGFRHPRNJZJHLQZDGRGOWYFEZYIZT2NQ, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.861350291 (151h11m38.557199034s) [noise=true , address=145.79.227.31:4797 , piece_id=2TFSZNURIAU573QUA5DUS7AJZNY2OXC6TJL3ST4VXIWY2WVB5YJA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.863368665 (151h11m38.559217418s) [noise=true , address=145.79.227.23:4547 , piece_id=K54H437KW5AAYUWOTMDBDP5GJ3ZABFKLAAOSS6WS5G5CROYEHODQ, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.864848794 (151h11m38.560697547s) [noise=true , address=145.79.227.24:4581 , piece_id=TSCVCH2KJEVMD4AVOEIHGTMRCRZ6NZMZA2CTQ6FUO2EIL3JAVYEA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.865644930 (151h11m38.561493683s) [noise=true , address=145.79.227.8:4330 , piece_id=4A47HIUX2MTJ2673XZZCMMF4T235TS4JWMNYJEVHHCKAHQX6APMA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:08.560504137 (151h11m40.25635289s) [upload=133242]: Next returned nil segment Nov 5 18:52:08.560522693 (151h11m40.256371446s) [upload=133242]: Waiting for error group managing segments... Nov 5 18:52:08.850333179 (151h11m40.546181932s) [noise=true , address=145.79.227.17:4972 , piece_id=H7UHQV7RDLCZUHY2XMXUAD2PX4ZBLWKTM47N6PH7UOGBRXPMWVOA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.853517822 (151h11m40.549366575s) [noise=true , address=145.79.227.22:4509 , piece_id=7RA5NJ72PI3DT4IC2ECVLZ3A2HXXL546ELVDXHZW4OXNN4D36OEA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.855547508 (151h11m40.551396261s) [noise=true , address=145.79.227.9:4366 , piece_id=7ZRECKBQC4S6QLERBZY442WYS4H35XWOZYSGGPHSU7NQ6WL4UNVA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.856103106 (151h11m40.551951859s) [noise=true , address=145.79.227.25:4618 , piece_id=XRUPY5BUPG7BXN2MI6MGYO4N3GW6TZDJE46MZFNRELWANHHG3N5A, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.860417321 (151h11m40.556266064s) [noise=true , address=145.79.227.8:4330 , piece_id=4A47HIUX2MTJ2673XZZCMMF4T235TS4JWMNYJEVHHCKAHQX6APMA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.865602796 (151h11m40.561451549s) [noise=true , address=145.79.227.23:4547 , piece_id=K54H437KW5AAYUWOTMDBDP5GJ3ZABFKLAAOSS6WS5G5CROYEHODQ, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.879393757 (151h11m40.57524251s) [noise=true , address=145.79.227.31:4797 , piece_id=2TFSZNURIAU573QUA5DUS7AJZNY2OXC6TJL3ST4VXIWY2WVB5YJA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.905541449 (151h11m40.601390202s) [noise=true , address=145.79.227.26:4654 , piece_id=E5J22KNPQSETSY443QQZXFFKUUJFHWFV22SQAVZ4BFQ4M6YWCNQQ, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.929018836 (151h11m40.624867589s) [noise=true , address=145.79.227.24:4581 , piece_id=TSCVCH2KJEVMD4AVOEIHGTMRCRZ6NZMZA2CTQ6FUO2EIL3JAVYEA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:35.899554917 (151h12m7.59540368s) [noise=true , address=77.90.31.17:4771 , piece_id=KVFE7VYW3RZJ25BWSH4BIRRQCMWDYOIK7EJ62XVREGN26YN7XYMQ, seg_pos={0 0}, upload=133242]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: write tcp 188.166.225.170:60676->77.90.31.17:4771: use of closed network connection; write tcp 188.166.225.170:60676->77.90.31.17:4771: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60676->77.90.31.17:4771: use of closed network connection Nov 5 18:52:35.899727506 (151h12m7.595576259s) [noise=true , address=77.90.31.13:4628 , piece_id=QFTEHZ5FY7A5DPZZ44TRXGFRHPRNJZJHLQZDGRGOWYFEZYIZT2NQ, seg_pos={0 0}, upload=133242]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: write tcp 188.166.225.170:42026->77.90.31.13:4628: use of closed network connection; write tcp 188.166.225.170:42026->77.90.31.13:4628: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42026->77.90.31.13:4628: use of closed network connection Nov 5 18:52:35.899860439 (151h12m7.595709192s) [noise=true , address=45.91.116.7:4010 , piece_id=EBVGGNQF3N4MTUIATJUWK4QZN7SEOWI6SIAW6N7JZOB5G6H33NEQ, seg_pos={0 0}, upload=133242]: Done uploading piece. err: ecclient: upload failed (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, address:45.91.116.7:4010): protocol: write tcp 188.166.225.170:40036->45.91.116.7:4010: use of closed network connection; write tcp 188.166.225.170:40036->45.91.116.7:4010: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40036->45.91.116.7:4010: use of closed network connection Nov 5 18:52:35.899923829 (151h12m7.595772572s) [noise=true , address=77.90.31.18:4805 , piece_id=U2U34KK4AJK4A5HGNTHEYCIARLAR4X5NLOVCF63TN3PJUIRMMIHA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: write tcp 188.166.225.170:60758->77.90.31.18:4805: use of closed network connection; write tcp 188.166.225.170:60758->77.90.31.18:4805: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60758->77.90.31.18:4805: use of closed network connection Nov 5 18:52:35.900052084 (151h12m7.595900837s) [noise=true , address=77.90.31.16:4734 , piece_id=HFWVFM46PZFLZT6YNZVIA6HKLVN2PTV2GETZEKX356DMDGQO4HOA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: write tcp 188.166.225.170:50728->77.90.31.16:4734: use of closed network connection; write tcp 188.166.225.170:50728->77.90.31.16:4734: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50728->77.90.31.16:4734: use of closed network connection Nov 5 18:52:35.900231175 (151h12m7.596079938s) [noise=true , address=45.91.116.11:4117 , piece_id=QUGUYHEJWEEV733CBC3ONMV7XWDKVH2OFLQWXPQHPKZ7GTH4RLUA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: ecclient: upload failed (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, address:45.91.116.11:4117): protocol: write tcp 188.166.225.170:41292->45.91.116.11:4117: use of closed network connection; write tcp 188.166.225.170:41292->45.91.116.11:4117: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41292->45.91.116.11:4117: use of closed network connection Nov 5 18:52:35.900342116 (151h12m7.596190859s) [noise=true , address=77.90.31.15:4698 , piece_id=7VACL26N7ZCBXIA2I3XL5Q3UJ2BAHDIPXX4VZCQX3S63ABNW625Q, seg_pos={0 0}, upload=133242]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: write tcp 188.166.225.170:41118->77.90.31.15:4698: use of closed network connection; write tcp 188.166.225.170:41118->77.90.31.15:4698: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41118->77.90.31.15:4698: use of closed network connection Nov 5 18:52:35.900382814 (151h12m7.596231567s) [seg_pos={0 0}, upload=133242]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Nov 5 18:52:36.162442033 (151h12m7.858290786s) [upload=133242]: Done uploading segments... Nov 5 18:52:36.162481568 (151h12m7.858330321s) [upload=133242]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 5.835912913s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [......XX...X................................................] duration: 3.471887763s trace: https://jaeger.dev.storj.io/trace/204ab2570dc82d9c FAIL (2025-11-05 19:28:04 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.471923541s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [....X...X...................................................] duration: 5.913670784s trace: https://jaeger.dev.storj.io/trace/2e576aae48e8f03e FAIL (2025-11-05 19:27:51 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== 5 18:52:06.850167887 (151h11m38.54601663s) [noise=true , address=145.79.227.25:4618 , piece_id=XRUPY5BUPG7BXN2MI6MGYO4N3GW6TZDJE46MZFNRELWANHHG3N5A, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.850747301 (151h11m38.546596054s) [noise=true , address=77.90.31.15:4698 , piece_id=7VACL26N7ZCBXIA2I3XL5Q3UJ2BAHDIPXX4VZCQX3S63ABNW625Q, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.851222605 (151h11m38.547071358s) [noise=true , address=145.79.227.17:4972 , piece_id=H7UHQV7RDLCZUHY2XMXUAD2PX4ZBLWKTM47N6PH7UOGBRXPMWVOA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.851558575 (151h11m38.547407319s) [noise=true , address=145.79.227.22:4509 , piece_id=7RA5NJ72PI3DT4IC2ECVLZ3A2HXXL546ELVDXHZW4OXNN4D36OEA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.851909263 (151h11m38.547758006s) [noise=true , address=145.79.227.26:4654 , piece_id=E5J22KNPQSETSY443QQZXFFKUUJFHWFV22SQAVZ4BFQ4M6YWCNQQ, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.852327540 (151h11m38.548176293s) [noise=true , address=45.91.116.7:4010 , piece_id=EBVGGNQF3N4MTUIATJUWK4QZN7SEOWI6SIAW6N7JZOB5G6H33NEQ, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.852728624 (151h11m38.548577377s) [noise=true , address=145.79.227.9:4366 , piece_id=7ZRECKBQC4S6QLERBZY442WYS4H35XWOZYSGGPHSU7NQ6WL4UNVA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.853978774 (151h11m38.549827528s) [noise=true , address=77.90.31.18:4805 , piece_id=U2U34KK4AJK4A5HGNTHEYCIARLAR4X5NLOVCF63TN3PJUIRMMIHA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.854447296 (151h11m38.550296059s) [noise=true , address=77.90.31.16:4734 , piece_id=HFWVFM46PZFLZT6YNZVIA6HKLVN2PTV2GETZEKX356DMDGQO4HOA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.856523089 (151h11m38.552371842s) [noise=true , address=45.91.116.11:4117 , piece_id=QUGUYHEJWEEV733CBC3ONMV7XWDKVH2OFLQWXPQHPKZ7GTH4RLUA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.859490379 (151h11m38.555339122s) [noise=true , address=77.90.31.13:4628 , piece_id=QFTEHZ5FY7A5DPZZ44TRXGFRHPRNJZJHLQZDGRGOWYFEZYIZT2NQ, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.861350291 (151h11m38.557199034s) [noise=true , address=145.79.227.31:4797 , piece_id=2TFSZNURIAU573QUA5DUS7AJZNY2OXC6TJL3ST4VXIWY2WVB5YJA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.863368665 (151h11m38.559217418s) [noise=true , address=145.79.227.23:4547 , piece_id=K54H437KW5AAYUWOTMDBDP5GJ3ZABFKLAAOSS6WS5G5CROYEHODQ, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.864848794 (151h11m38.560697547s) [noise=true , address=145.79.227.24:4581 , piece_id=TSCVCH2KJEVMD4AVOEIHGTMRCRZ6NZMZA2CTQ6FUO2EIL3JAVYEA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.865644930 (151h11m38.561493683s) [noise=true , address=145.79.227.8:4330 , piece_id=4A47HIUX2MTJ2673XZZCMMF4T235TS4JWMNYJEVHHCKAHQX6APMA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:08.560504137 (151h11m40.25635289s) [upload=133242]: Next returned nil segment Nov 5 18:52:08.560522693 (151h11m40.256371446s) [upload=133242]: Waiting for error group managing segments... Nov 5 18:52:08.850333179 (151h11m40.546181932s) [noise=true , address=145.79.227.17:4972 , piece_id=H7UHQV7RDLCZUHY2XMXUAD2PX4ZBLWKTM47N6PH7UOGBRXPMWVOA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.853517822 (151h11m40.549366575s) [noise=true , address=145.79.227.22:4509 , piece_id=7RA5NJ72PI3DT4IC2ECVLZ3A2HXXL546ELVDXHZW4OXNN4D36OEA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.855547508 (151h11m40.551396261s) [noise=true , address=145.79.227.9:4366 , piece_id=7ZRECKBQC4S6QLERBZY442WYS4H35XWOZYSGGPHSU7NQ6WL4UNVA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.856103106 (151h11m40.551951859s) [noise=true , address=145.79.227.25:4618 , piece_id=XRUPY5BUPG7BXN2MI6MGYO4N3GW6TZDJE46MZFNRELWANHHG3N5A, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.860417321 (151h11m40.556266064s) [noise=true , address=145.79.227.8:4330 , piece_id=4A47HIUX2MTJ2673XZZCMMF4T235TS4JWMNYJEVHHCKAHQX6APMA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.865602796 (151h11m40.561451549s) [noise=true , address=145.79.227.23:4547 , piece_id=K54H437KW5AAYUWOTMDBDP5GJ3ZABFKLAAOSS6WS5G5CROYEHODQ, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.879393757 (151h11m40.57524251s) [noise=true , address=145.79.227.31:4797 , piece_id=2TFSZNURIAU573QUA5DUS7AJZNY2OXC6TJL3ST4VXIWY2WVB5YJA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.905541449 (151h11m40.601390202s) [noise=true , address=145.79.227.26:4654 , piece_id=E5J22KNPQSETSY443QQZXFFKUUJFHWFV22SQAVZ4BFQ4M6YWCNQQ, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.929018836 (151h11m40.624867589s) [noise=true , address=145.79.227.24:4581 , piece_id=TSCVCH2KJEVMD4AVOEIHGTMRCRZ6NZMZA2CTQ6FUO2EIL3JAVYEA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:35.899554917 (151h12m7.59540368s) [noise=true , address=77.90.31.17:4771 , piece_id=KVFE7VYW3RZJ25BWSH4BIRRQCMWDYOIK7EJ62XVREGN26YN7XYMQ, seg_pos={0 0}, upload=133242]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: write tcp 188.166.225.170:60676->77.90.31.17:4771: use of closed network connection; write tcp 188.166.225.170:60676->77.90.31.17:4771: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60676->77.90.31.17:4771: use of closed network connection Nov 5 18:52:35.899727506 (151h12m7.595576259s) [noise=true , address=77.90.31.13:4628 , piece_id=QFTEHZ5FY7A5DPZZ44TRXGFRHPRNJZJHLQZDGRGOWYFEZYIZT2NQ, seg_pos={0 0}, upload=133242]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: write tcp 188.166.225.170:42026->77.90.31.13:4628: use of closed network connection; write tcp 188.166.225.170:42026->77.90.31.13:4628: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42026->77.90.31.13:4628: use of closed network connection Nov 5 18:52:35.899860439 (151h12m7.595709192s) [noise=true , address=45.91.116.7:4010 , piece_id=EBVGGNQF3N4MTUIATJUWK4QZN7SEOWI6SIAW6N7JZOB5G6H33NEQ, seg_pos={0 0}, upload=133242]: Done uploading piece. err: ecclient: upload failed (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, address:45.91.116.7:4010): protocol: write tcp 188.166.225.170:40036->45.91.116.7:4010: use of closed network connection; write tcp 188.166.225.170:40036->45.91.116.7:4010: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40036->45.91.116.7:4010: use of closed network connection Nov 5 18:52:35.899923829 (151h12m7.595772572s) [noise=true , address=77.90.31.18:4805 , piece_id=U2U34KK4AJK4A5HGNTHEYCIARLAR4X5NLOVCF63TN3PJUIRMMIHA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: write tcp 188.166.225.170:60758->77.90.31.18:4805: use of closed network connection; write tcp 188.166.225.170:60758->77.90.31.18:4805: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60758->77.90.31.18:4805: use of closed network connection Nov 5 18:52:35.900052084 (151h12m7.595900837s) [noise=true , address=77.90.31.16:4734 , piece_id=HFWVFM46PZFLZT6YNZVIA6HKLVN2PTV2GETZEKX356DMDGQO4HOA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: write tcp 188.166.225.170:50728->77.90.31.16:4734: use of closed network connection; write tcp 188.166.225.170:50728->77.90.31.16:4734: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50728->77.90.31.16:4734: use of closed network connection Nov 5 18:52:35.900231175 (151h12m7.596079938s) [noise=true , address=45.91.116.11:4117 , piece_id=QUGUYHEJWEEV733CBC3ONMV7XWDKVH2OFLQWXPQHPKZ7GTH4RLUA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: ecclient: upload failed (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, address:45.91.116.11:4117): protocol: write tcp 188.166.225.170:41292->45.91.116.11:4117: use of closed network connection; write tcp 188.166.225.170:41292->45.91.116.11:4117: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41292->45.91.116.11:4117: use of closed network connection Nov 5 18:52:35.900342116 (151h12m7.596190859s) [noise=true , address=77.90.31.15:4698 , piece_id=7VACL26N7ZCBXIA2I3XL5Q3UJ2BAHDIPXX4VZCQX3S63ABNW625Q, seg_pos={0 0}, upload=133242]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: write tcp 188.166.225.170:41118->77.90.31.15:4698: use of closed network connection; write tcp 188.166.225.170:41118->77.90.31.15:4698: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41118->77.90.31.15:4698: use of closed network connection Nov 5 18:52:35.900382814 (151h12m7.596231567s) [seg_pos={0 0}, upload=133242]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Nov 5 18:52:36.162442033 (151h12m7.858290786s) [upload=133242]: Done uploading segments... Nov 5 18:52:36.162481568 (151h12m7.858330321s) [upload=133242]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 5.913693236s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [.......XX...X...............................................] duration: 3.488144941s trace: https://jaeger.dev.storj.io/trace/204ab2570dc82d9c FAIL (2025-11-05 19:27:46 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.488172785s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [......X...X.................................................] duration: 6.07793323s trace: https://jaeger.dev.storj.io/trace/74cf1e30f6d49950 FAIL (2025-11-05 19:27:42 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.077961022s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [........X..X................................................] duration: 5.749105459s trace: https://jaeger.dev.storj.io/trace/3e2ae8884260b157 FAIL (2025-11-05 19:27:39 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.74913204s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....X...X..................................................] duration: 5.893182761s trace: https://jaeger.dev.storj.io/trace/2e576aae48e8f03e FAIL (2025-11-05 19:27:35 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== 5 18:52:06.850167887 (151h11m38.54601663s) [noise=true , address=145.79.227.25:4618 , piece_id=XRUPY5BUPG7BXN2MI6MGYO4N3GW6TZDJE46MZFNRELWANHHG3N5A, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.850747301 (151h11m38.546596054s) [noise=true , address=77.90.31.15:4698 , piece_id=7VACL26N7ZCBXIA2I3XL5Q3UJ2BAHDIPXX4VZCQX3S63ABNW625Q, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.851222605 (151h11m38.547071358s) [noise=true , address=145.79.227.17:4972 , piece_id=H7UHQV7RDLCZUHY2XMXUAD2PX4ZBLWKTM47N6PH7UOGBRXPMWVOA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.851558575 (151h11m38.547407319s) [noise=true , address=145.79.227.22:4509 , piece_id=7RA5NJ72PI3DT4IC2ECVLZ3A2HXXL546ELVDXHZW4OXNN4D36OEA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.851909263 (151h11m38.547758006s) [noise=true , address=145.79.227.26:4654 , piece_id=E5J22KNPQSETSY443QQZXFFKUUJFHWFV22SQAVZ4BFQ4M6YWCNQQ, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.852327540 (151h11m38.548176293s) [noise=true , address=45.91.116.7:4010 , piece_id=EBVGGNQF3N4MTUIATJUWK4QZN7SEOWI6SIAW6N7JZOB5G6H33NEQ, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.852728624 (151h11m38.548577377s) [noise=true , address=145.79.227.9:4366 , piece_id=7ZRECKBQC4S6QLERBZY442WYS4H35XWOZYSGGPHSU7NQ6WL4UNVA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.853978774 (151h11m38.549827528s) [noise=true , address=77.90.31.18:4805 , piece_id=U2U34KK4AJK4A5HGNTHEYCIARLAR4X5NLOVCF63TN3PJUIRMMIHA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.854447296 (151h11m38.550296059s) [noise=true , address=77.90.31.16:4734 , piece_id=HFWVFM46PZFLZT6YNZVIA6HKLVN2PTV2GETZEKX356DMDGQO4HOA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.856523089 (151h11m38.552371842s) [noise=true , address=45.91.116.11:4117 , piece_id=QUGUYHEJWEEV733CBC3ONMV7XWDKVH2OFLQWXPQHPKZ7GTH4RLUA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.859490379 (151h11m38.555339122s) [noise=true , address=77.90.31.13:4628 , piece_id=QFTEHZ5FY7A5DPZZ44TRXGFRHPRNJZJHLQZDGRGOWYFEZYIZT2NQ, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.861350291 (151h11m38.557199034s) [noise=true , address=145.79.227.31:4797 , piece_id=2TFSZNURIAU573QUA5DUS7AJZNY2OXC6TJL3ST4VXIWY2WVB5YJA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.863368665 (151h11m38.559217418s) [noise=true , address=145.79.227.23:4547 , piece_id=K54H437KW5AAYUWOTMDBDP5GJ3ZABFKLAAOSS6WS5G5CROYEHODQ, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.864848794 (151h11m38.560697547s) [noise=true , address=145.79.227.24:4581 , piece_id=TSCVCH2KJEVMD4AVOEIHGTMRCRZ6NZMZA2CTQ6FUO2EIL3JAVYEA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.865644930 (151h11m38.561493683s) [noise=true , address=145.79.227.8:4330 , piece_id=4A47HIUX2MTJ2673XZZCMMF4T235TS4JWMNYJEVHHCKAHQX6APMA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:08.560504137 (151h11m40.25635289s) [upload=133242]: Next returned nil segment Nov 5 18:52:08.560522693 (151h11m40.256371446s) [upload=133242]: Waiting for error group managing segments... Nov 5 18:52:08.850333179 (151h11m40.546181932s) [noise=true , address=145.79.227.17:4972 , piece_id=H7UHQV7RDLCZUHY2XMXUAD2PX4ZBLWKTM47N6PH7UOGBRXPMWVOA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.853517822 (151h11m40.549366575s) [noise=true , address=145.79.227.22:4509 , piece_id=7RA5NJ72PI3DT4IC2ECVLZ3A2HXXL546ELVDXHZW4OXNN4D36OEA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.855547508 (151h11m40.551396261s) [noise=true , address=145.79.227.9:4366 , piece_id=7ZRECKBQC4S6QLERBZY442WYS4H35XWOZYSGGPHSU7NQ6WL4UNVA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.856103106 (151h11m40.551951859s) [noise=true , address=145.79.227.25:4618 , piece_id=XRUPY5BUPG7BXN2MI6MGYO4N3GW6TZDJE46MZFNRELWANHHG3N5A, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.860417321 (151h11m40.556266064s) [noise=true , address=145.79.227.8:4330 , piece_id=4A47HIUX2MTJ2673XZZCMMF4T235TS4JWMNYJEVHHCKAHQX6APMA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.865602796 (151h11m40.561451549s) [noise=true , address=145.79.227.23:4547 , piece_id=K54H437KW5AAYUWOTMDBDP5GJ3ZABFKLAAOSS6WS5G5CROYEHODQ, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.879393757 (151h11m40.57524251s) [noise=true , address=145.79.227.31:4797 , piece_id=2TFSZNURIAU573QUA5DUS7AJZNY2OXC6TJL3ST4VXIWY2WVB5YJA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.905541449 (151h11m40.601390202s) [noise=true , address=145.79.227.26:4654 , piece_id=E5J22KNPQSETSY443QQZXFFKUUJFHWFV22SQAVZ4BFQ4M6YWCNQQ, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.929018836 (151h11m40.624867589s) [noise=true , address=145.79.227.24:4581 , piece_id=TSCVCH2KJEVMD4AVOEIHGTMRCRZ6NZMZA2CTQ6FUO2EIL3JAVYEA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:35.899554917 (151h12m7.59540368s) [noise=true , address=77.90.31.17:4771 , piece_id=KVFE7VYW3RZJ25BWSH4BIRRQCMWDYOIK7EJ62XVREGN26YN7XYMQ, seg_pos={0 0}, upload=133242]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: write tcp 188.166.225.170:60676->77.90.31.17:4771: use of closed network connection; write tcp 188.166.225.170:60676->77.90.31.17:4771: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60676->77.90.31.17:4771: use of closed network connection Nov 5 18:52:35.899727506 (151h12m7.595576259s) [noise=true , address=77.90.31.13:4628 , piece_id=QFTEHZ5FY7A5DPZZ44TRXGFRHPRNJZJHLQZDGRGOWYFEZYIZT2NQ, seg_pos={0 0}, upload=133242]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: write tcp 188.166.225.170:42026->77.90.31.13:4628: use of closed network connection; write tcp 188.166.225.170:42026->77.90.31.13:4628: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42026->77.90.31.13:4628: use of closed network connection Nov 5 18:52:35.899860439 (151h12m7.595709192s) [noise=true , address=45.91.116.7:4010 , piece_id=EBVGGNQF3N4MTUIATJUWK4QZN7SEOWI6SIAW6N7JZOB5G6H33NEQ, seg_pos={0 0}, upload=133242]: Done uploading piece. err: ecclient: upload failed (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, address:45.91.116.7:4010): protocol: write tcp 188.166.225.170:40036->45.91.116.7:4010: use of closed network connection; write tcp 188.166.225.170:40036->45.91.116.7:4010: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40036->45.91.116.7:4010: use of closed network connection Nov 5 18:52:35.899923829 (151h12m7.595772572s) [noise=true , address=77.90.31.18:4805 , piece_id=U2U34KK4AJK4A5HGNTHEYCIARLAR4X5NLOVCF63TN3PJUIRMMIHA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: write tcp 188.166.225.170:60758->77.90.31.18:4805: use of closed network connection; write tcp 188.166.225.170:60758->77.90.31.18:4805: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60758->77.90.31.18:4805: use of closed network connection Nov 5 18:52:35.900052084 (151h12m7.595900837s) [noise=true , address=77.90.31.16:4734 , piece_id=HFWVFM46PZFLZT6YNZVIA6HKLVN2PTV2GETZEKX356DMDGQO4HOA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: write tcp 188.166.225.170:50728->77.90.31.16:4734: use of closed network connection; write tcp 188.166.225.170:50728->77.90.31.16:4734: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50728->77.90.31.16:4734: use of closed network connection Nov 5 18:52:35.900231175 (151h12m7.596079938s) [noise=true , address=45.91.116.11:4117 , piece_id=QUGUYHEJWEEV733CBC3ONMV7XWDKVH2OFLQWXPQHPKZ7GTH4RLUA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: ecclient: upload failed (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, address:45.91.116.11:4117): protocol: write tcp 188.166.225.170:41292->45.91.116.11:4117: use of closed network connection; write tcp 188.166.225.170:41292->45.91.116.11:4117: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41292->45.91.116.11:4117: use of closed network connection Nov 5 18:52:35.900342116 (151h12m7.596190859s) [noise=true , address=77.90.31.15:4698 , piece_id=7VACL26N7ZCBXIA2I3XL5Q3UJ2BAHDIPXX4VZCQX3S63ABNW625Q, seg_pos={0 0}, upload=133242]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: write tcp 188.166.225.170:41118->77.90.31.15:4698: use of closed network connection; write tcp 188.166.225.170:41118->77.90.31.15:4698: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41118->77.90.31.15:4698: use of closed network connection Nov 5 18:52:35.900382814 (151h12m7.596231567s) [seg_pos={0 0}, upload=133242]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Nov 5 18:52:36.162442033 (151h12m7.858290786s) [upload=133242]: Done uploading segments... Nov 5 18:52:36.162481568 (151h12m7.858330321s) [upload=133242]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 5.893211906s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [........XX...X..............................................] duration: 3.484453119s trace: https://jaeger.dev.storj.io/trace/204ab2570dc82d9c FAIL (2025-11-05 19:27:30 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.484486782s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.........X..X...............................................] duration: 5.714243678s trace: https://jaeger.dev.storj.io/trace/3e2ae8884260b157 FAIL (2025-11-05 19:27:11 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.71427105s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [.........XX...X.............................................] duration: 3.448696012s trace: https://jaeger.dev.storj.io/trace/204ab2570dc82d9c FAIL (2025-11-05 19:26:54 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.448725348s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......X...X................................................] duration: 6.329283766s trace: https://jaeger.dev.storj.io/trace/74cf1e30f6d49950 FAIL (2025-11-05 19:26:52 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.329313173s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [......X...X.................................................] duration: 5.998331881s trace: https://jaeger.dev.storj.io/trace/2e576aae48e8f03e FAIL (2025-11-05 19:26:49 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== 5 18:52:06.850167887 (151h11m38.54601663s) [noise=true , address=145.79.227.25:4618 , piece_id=XRUPY5BUPG7BXN2MI6MGYO4N3GW6TZDJE46MZFNRELWANHHG3N5A, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.850747301 (151h11m38.546596054s) [noise=true , address=77.90.31.15:4698 , piece_id=7VACL26N7ZCBXIA2I3XL5Q3UJ2BAHDIPXX4VZCQX3S63ABNW625Q, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.851222605 (151h11m38.547071358s) [noise=true , address=145.79.227.17:4972 , piece_id=H7UHQV7RDLCZUHY2XMXUAD2PX4ZBLWKTM47N6PH7UOGBRXPMWVOA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.851558575 (151h11m38.547407319s) [noise=true , address=145.79.227.22:4509 , piece_id=7RA5NJ72PI3DT4IC2ECVLZ3A2HXXL546ELVDXHZW4OXNN4D36OEA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.851909263 (151h11m38.547758006s) [noise=true , address=145.79.227.26:4654 , piece_id=E5J22KNPQSETSY443QQZXFFKUUJFHWFV22SQAVZ4BFQ4M6YWCNQQ, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.852327540 (151h11m38.548176293s) [noise=true , address=45.91.116.7:4010 , piece_id=EBVGGNQF3N4MTUIATJUWK4QZN7SEOWI6SIAW6N7JZOB5G6H33NEQ, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.852728624 (151h11m38.548577377s) [noise=true , address=145.79.227.9:4366 , piece_id=7ZRECKBQC4S6QLERBZY442WYS4H35XWOZYSGGPHSU7NQ6WL4UNVA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.853978774 (151h11m38.549827528s) [noise=true , address=77.90.31.18:4805 , piece_id=U2U34KK4AJK4A5HGNTHEYCIARLAR4X5NLOVCF63TN3PJUIRMMIHA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.854447296 (151h11m38.550296059s) [noise=true , address=77.90.31.16:4734 , piece_id=HFWVFM46PZFLZT6YNZVIA6HKLVN2PTV2GETZEKX356DMDGQO4HOA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.856523089 (151h11m38.552371842s) [noise=true , address=45.91.116.11:4117 , piece_id=QUGUYHEJWEEV733CBC3ONMV7XWDKVH2OFLQWXPQHPKZ7GTH4RLUA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.859490379 (151h11m38.555339122s) [noise=true , address=77.90.31.13:4628 , piece_id=QFTEHZ5FY7A5DPZZ44TRXGFRHPRNJZJHLQZDGRGOWYFEZYIZT2NQ, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.861350291 (151h11m38.557199034s) [noise=true , address=145.79.227.31:4797 , piece_id=2TFSZNURIAU573QUA5DUS7AJZNY2OXC6TJL3ST4VXIWY2WVB5YJA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.863368665 (151h11m38.559217418s) [noise=true , address=145.79.227.23:4547 , piece_id=K54H437KW5AAYUWOTMDBDP5GJ3ZABFKLAAOSS6WS5G5CROYEHODQ, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.864848794 (151h11m38.560697547s) [noise=true , address=145.79.227.24:4581 , piece_id=TSCVCH2KJEVMD4AVOEIHGTMRCRZ6NZMZA2CTQ6FUO2EIL3JAVYEA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.865644930 (151h11m38.561493683s) [noise=true , address=145.79.227.8:4330 , piece_id=4A47HIUX2MTJ2673XZZCMMF4T235TS4JWMNYJEVHHCKAHQX6APMA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:08.560504137 (151h11m40.25635289s) [upload=133242]: Next returned nil segment Nov 5 18:52:08.560522693 (151h11m40.256371446s) [upload=133242]: Waiting for error group managing segments... Nov 5 18:52:08.850333179 (151h11m40.546181932s) [noise=true , address=145.79.227.17:4972 , piece_id=H7UHQV7RDLCZUHY2XMXUAD2PX4ZBLWKTM47N6PH7UOGBRXPMWVOA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.853517822 (151h11m40.549366575s) [noise=true , address=145.79.227.22:4509 , piece_id=7RA5NJ72PI3DT4IC2ECVLZ3A2HXXL546ELVDXHZW4OXNN4D36OEA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.855547508 (151h11m40.551396261s) [noise=true , address=145.79.227.9:4366 , piece_id=7ZRECKBQC4S6QLERBZY442WYS4H35XWOZYSGGPHSU7NQ6WL4UNVA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.856103106 (151h11m40.551951859s) [noise=true , address=145.79.227.25:4618 , piece_id=XRUPY5BUPG7BXN2MI6MGYO4N3GW6TZDJE46MZFNRELWANHHG3N5A, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.860417321 (151h11m40.556266064s) [noise=true , address=145.79.227.8:4330 , piece_id=4A47HIUX2MTJ2673XZZCMMF4T235TS4JWMNYJEVHHCKAHQX6APMA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.865602796 (151h11m40.561451549s) [noise=true , address=145.79.227.23:4547 , piece_id=K54H437KW5AAYUWOTMDBDP5GJ3ZABFKLAAOSS6WS5G5CROYEHODQ, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.879393757 (151h11m40.57524251s) [noise=true , address=145.79.227.31:4797 , piece_id=2TFSZNURIAU573QUA5DUS7AJZNY2OXC6TJL3ST4VXIWY2WVB5YJA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.905541449 (151h11m40.601390202s) [noise=true , address=145.79.227.26:4654 , piece_id=E5J22KNPQSETSY443QQZXFFKUUJFHWFV22SQAVZ4BFQ4M6YWCNQQ, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.929018836 (151h11m40.624867589s) [noise=true , address=145.79.227.24:4581 , piece_id=TSCVCH2KJEVMD4AVOEIHGTMRCRZ6NZMZA2CTQ6FUO2EIL3JAVYEA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:35.899554917 (151h12m7.59540368s) [noise=true , address=77.90.31.17:4771 , piece_id=KVFE7VYW3RZJ25BWSH4BIRRQCMWDYOIK7EJ62XVREGN26YN7XYMQ, seg_pos={0 0}, upload=133242]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: write tcp 188.166.225.170:60676->77.90.31.17:4771: use of closed network connection; write tcp 188.166.225.170:60676->77.90.31.17:4771: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60676->77.90.31.17:4771: use of closed network connection Nov 5 18:52:35.899727506 (151h12m7.595576259s) [noise=true , address=77.90.31.13:4628 , piece_id=QFTEHZ5FY7A5DPZZ44TRXGFRHPRNJZJHLQZDGRGOWYFEZYIZT2NQ, seg_pos={0 0}, upload=133242]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: write tcp 188.166.225.170:42026->77.90.31.13:4628: use of closed network connection; write tcp 188.166.225.170:42026->77.90.31.13:4628: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42026->77.90.31.13:4628: use of closed network connection Nov 5 18:52:35.899860439 (151h12m7.595709192s) [noise=true , address=45.91.116.7:4010 , piece_id=EBVGGNQF3N4MTUIATJUWK4QZN7SEOWI6SIAW6N7JZOB5G6H33NEQ, seg_pos={0 0}, upload=133242]: Done uploading piece. err: ecclient: upload failed (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, address:45.91.116.7:4010): protocol: write tcp 188.166.225.170:40036->45.91.116.7:4010: use of closed network connection; write tcp 188.166.225.170:40036->45.91.116.7:4010: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40036->45.91.116.7:4010: use of closed network connection Nov 5 18:52:35.899923829 (151h12m7.595772572s) [noise=true , address=77.90.31.18:4805 , piece_id=U2U34KK4AJK4A5HGNTHEYCIARLAR4X5NLOVCF63TN3PJUIRMMIHA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: write tcp 188.166.225.170:60758->77.90.31.18:4805: use of closed network connection; write tcp 188.166.225.170:60758->77.90.31.18:4805: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60758->77.90.31.18:4805: use of closed network connection Nov 5 18:52:35.900052084 (151h12m7.595900837s) [noise=true , address=77.90.31.16:4734 , piece_id=HFWVFM46PZFLZT6YNZVIA6HKLVN2PTV2GETZEKX356DMDGQO4HOA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: write tcp 188.166.225.170:50728->77.90.31.16:4734: use of closed network connection; write tcp 188.166.225.170:50728->77.90.31.16:4734: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50728->77.90.31.16:4734: use of closed network connection Nov 5 18:52:35.900231175 (151h12m7.596079938s) [noise=true , address=45.91.116.11:4117 , piece_id=QUGUYHEJWEEV733CBC3ONMV7XWDKVH2OFLQWXPQHPKZ7GTH4RLUA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: ecclient: upload failed (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, address:45.91.116.11:4117): protocol: write tcp 188.166.225.170:41292->45.91.116.11:4117: use of closed network connection; write tcp 188.166.225.170:41292->45.91.116.11:4117: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41292->45.91.116.11:4117: use of closed network connection Nov 5 18:52:35.900342116 (151h12m7.596190859s) [noise=true , address=77.90.31.15:4698 , piece_id=7VACL26N7ZCBXIA2I3XL5Q3UJ2BAHDIPXX4VZCQX3S63ABNW625Q, seg_pos={0 0}, upload=133242]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: write tcp 188.166.225.170:41118->77.90.31.15:4698: use of closed network connection; write tcp 188.166.225.170:41118->77.90.31.15:4698: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41118->77.90.31.15:4698: use of closed network connection Nov 5 18:52:35.900382814 (151h12m7.596231567s) [seg_pos={0 0}, upload=133242]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Nov 5 18:52:36.162442033 (151h12m7.858290786s) [upload=133242]: Done uploading segments... Nov 5 18:52:36.162481568 (151h12m7.858330321s) [upload=133242]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 5.998381847s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [..........X..X..............................................] duration: 5.704187501s trace: https://jaeger.dev.storj.io/trace/3e2ae8884260b157 FAIL (2025-11-05 19:26:23 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.704218548s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......X...X................................................] duration: 5.962925213s trace: https://jaeger.dev.storj.io/trace/2e576aae48e8f03e FAIL (2025-11-05 19:26:13 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== 5 18:52:06.850167887 (151h11m38.54601663s) [noise=true , address=145.79.227.25:4618 , piece_id=XRUPY5BUPG7BXN2MI6MGYO4N3GW6TZDJE46MZFNRELWANHHG3N5A, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.850747301 (151h11m38.546596054s) [noise=true , address=77.90.31.15:4698 , piece_id=7VACL26N7ZCBXIA2I3XL5Q3UJ2BAHDIPXX4VZCQX3S63ABNW625Q, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.851222605 (151h11m38.547071358s) [noise=true , address=145.79.227.17:4972 , piece_id=H7UHQV7RDLCZUHY2XMXUAD2PX4ZBLWKTM47N6PH7UOGBRXPMWVOA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.851558575 (151h11m38.547407319s) [noise=true , address=145.79.227.22:4509 , piece_id=7RA5NJ72PI3DT4IC2ECVLZ3A2HXXL546ELVDXHZW4OXNN4D36OEA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.851909263 (151h11m38.547758006s) [noise=true , address=145.79.227.26:4654 , piece_id=E5J22KNPQSETSY443QQZXFFKUUJFHWFV22SQAVZ4BFQ4M6YWCNQQ, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.852327540 (151h11m38.548176293s) [noise=true , address=45.91.116.7:4010 , piece_id=EBVGGNQF3N4MTUIATJUWK4QZN7SEOWI6SIAW6N7JZOB5G6H33NEQ, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.852728624 (151h11m38.548577377s) [noise=true , address=145.79.227.9:4366 , piece_id=7ZRECKBQC4S6QLERBZY442WYS4H35XWOZYSGGPHSU7NQ6WL4UNVA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.853978774 (151h11m38.549827528s) [noise=true , address=77.90.31.18:4805 , piece_id=U2U34KK4AJK4A5HGNTHEYCIARLAR4X5NLOVCF63TN3PJUIRMMIHA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.854447296 (151h11m38.550296059s) [noise=true , address=77.90.31.16:4734 , piece_id=HFWVFM46PZFLZT6YNZVIA6HKLVN2PTV2GETZEKX356DMDGQO4HOA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.856523089 (151h11m38.552371842s) [noise=true , address=45.91.116.11:4117 , piece_id=QUGUYHEJWEEV733CBC3ONMV7XWDKVH2OFLQWXPQHPKZ7GTH4RLUA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.859490379 (151h11m38.555339122s) [noise=true , address=77.90.31.13:4628 , piece_id=QFTEHZ5FY7A5DPZZ44TRXGFRHPRNJZJHLQZDGRGOWYFEZYIZT2NQ, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.861350291 (151h11m38.557199034s) [noise=true , address=145.79.227.31:4797 , piece_id=2TFSZNURIAU573QUA5DUS7AJZNY2OXC6TJL3ST4VXIWY2WVB5YJA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.863368665 (151h11m38.559217418s) [noise=true , address=145.79.227.23:4547 , piece_id=K54H437KW5AAYUWOTMDBDP5GJ3ZABFKLAAOSS6WS5G5CROYEHODQ, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.864848794 (151h11m38.560697547s) [noise=true , address=145.79.227.24:4581 , piece_id=TSCVCH2KJEVMD4AVOEIHGTMRCRZ6NZMZA2CTQ6FUO2EIL3JAVYEA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.865644930 (151h11m38.561493683s) [noise=true , address=145.79.227.8:4330 , piece_id=4A47HIUX2MTJ2673XZZCMMF4T235TS4JWMNYJEVHHCKAHQX6APMA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:08.560504137 (151h11m40.25635289s) [upload=133242]: Next returned nil segment Nov 5 18:52:08.560522693 (151h11m40.256371446s) [upload=133242]: Waiting for error group managing segments... Nov 5 18:52:08.850333179 (151h11m40.546181932s) [noise=true , address=145.79.227.17:4972 , piece_id=H7UHQV7RDLCZUHY2XMXUAD2PX4ZBLWKTM47N6PH7UOGBRXPMWVOA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.853517822 (151h11m40.549366575s) [noise=true , address=145.79.227.22:4509 , piece_id=7RA5NJ72PI3DT4IC2ECVLZ3A2HXXL546ELVDXHZW4OXNN4D36OEA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.855547508 (151h11m40.551396261s) [noise=true , address=145.79.227.9:4366 , piece_id=7ZRECKBQC4S6QLERBZY442WYS4H35XWOZYSGGPHSU7NQ6WL4UNVA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.856103106 (151h11m40.551951859s) [noise=true , address=145.79.227.25:4618 , piece_id=XRUPY5BUPG7BXN2MI6MGYO4N3GW6TZDJE46MZFNRELWANHHG3N5A, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.860417321 (151h11m40.556266064s) [noise=true , address=145.79.227.8:4330 , piece_id=4A47HIUX2MTJ2673XZZCMMF4T235TS4JWMNYJEVHHCKAHQX6APMA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.865602796 (151h11m40.561451549s) [noise=true , address=145.79.227.23:4547 , piece_id=K54H437KW5AAYUWOTMDBDP5GJ3ZABFKLAAOSS6WS5G5CROYEHODQ, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.879393757 (151h11m40.57524251s) [noise=true , address=145.79.227.31:4797 , piece_id=2TFSZNURIAU573QUA5DUS7AJZNY2OXC6TJL3ST4VXIWY2WVB5YJA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.905541449 (151h11m40.601390202s) [noise=true , address=145.79.227.26:4654 , piece_id=E5J22KNPQSETSY443QQZXFFKUUJFHWFV22SQAVZ4BFQ4M6YWCNQQ, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.929018836 (151h11m40.624867589s) [noise=true , address=145.79.227.24:4581 , piece_id=TSCVCH2KJEVMD4AVOEIHGTMRCRZ6NZMZA2CTQ6FUO2EIL3JAVYEA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:35.899554917 (151h12m7.59540368s) [noise=true , address=77.90.31.17:4771 , piece_id=KVFE7VYW3RZJ25BWSH4BIRRQCMWDYOIK7EJ62XVREGN26YN7XYMQ, seg_pos={0 0}, upload=133242]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: write tcp 188.166.225.170:60676->77.90.31.17:4771: use of closed network connection; write tcp 188.166.225.170:60676->77.90.31.17:4771: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60676->77.90.31.17:4771: use of closed network connection Nov 5 18:52:35.899727506 (151h12m7.595576259s) [noise=true , address=77.90.31.13:4628 , piece_id=QFTEHZ5FY7A5DPZZ44TRXGFRHPRNJZJHLQZDGRGOWYFEZYIZT2NQ, seg_pos={0 0}, upload=133242]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: write tcp 188.166.225.170:42026->77.90.31.13:4628: use of closed network connection; write tcp 188.166.225.170:42026->77.90.31.13:4628: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42026->77.90.31.13:4628: use of closed network connection Nov 5 18:52:35.899860439 (151h12m7.595709192s) [noise=true , address=45.91.116.7:4010 , piece_id=EBVGGNQF3N4MTUIATJUWK4QZN7SEOWI6SIAW6N7JZOB5G6H33NEQ, seg_pos={0 0}, upload=133242]: Done uploading piece. err: ecclient: upload failed (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, address:45.91.116.7:4010): protocol: write tcp 188.166.225.170:40036->45.91.116.7:4010: use of closed network connection; write tcp 188.166.225.170:40036->45.91.116.7:4010: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40036->45.91.116.7:4010: use of closed network connection Nov 5 18:52:35.899923829 (151h12m7.595772572s) [noise=true , address=77.90.31.18:4805 , piece_id=U2U34KK4AJK4A5HGNTHEYCIARLAR4X5NLOVCF63TN3PJUIRMMIHA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: write tcp 188.166.225.170:60758->77.90.31.18:4805: use of closed network connection; write tcp 188.166.225.170:60758->77.90.31.18:4805: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60758->77.90.31.18:4805: use of closed network connection Nov 5 18:52:35.900052084 (151h12m7.595900837s) [noise=true , address=77.90.31.16:4734 , piece_id=HFWVFM46PZFLZT6YNZVIA6HKLVN2PTV2GETZEKX356DMDGQO4HOA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: write tcp 188.166.225.170:50728->77.90.31.16:4734: use of closed network connection; write tcp 188.166.225.170:50728->77.90.31.16:4734: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50728->77.90.31.16:4734: use of closed network connection Nov 5 18:52:35.900231175 (151h12m7.596079938s) [noise=true , address=45.91.116.11:4117 , piece_id=QUGUYHEJWEEV733CBC3ONMV7XWDKVH2OFLQWXPQHPKZ7GTH4RLUA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: ecclient: upload failed (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, address:45.91.116.11:4117): protocol: write tcp 188.166.225.170:41292->45.91.116.11:4117: use of closed network connection; write tcp 188.166.225.170:41292->45.91.116.11:4117: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41292->45.91.116.11:4117: use of closed network connection Nov 5 18:52:35.900342116 (151h12m7.596190859s) [noise=true , address=77.90.31.15:4698 , piece_id=7VACL26N7ZCBXIA2I3XL5Q3UJ2BAHDIPXX4VZCQX3S63ABNW625Q, seg_pos={0 0}, upload=133242]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: write tcp 188.166.225.170:41118->77.90.31.15:4698: use of closed network connection; write tcp 188.166.225.170:41118->77.90.31.15:4698: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41118->77.90.31.15:4698: use of closed network connection Nov 5 18:52:35.900382814 (151h12m7.596231567s) [seg_pos={0 0}, upload=133242]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Nov 5 18:52:36.162442033 (151h12m7.858290786s) [upload=133242]: Done uploading segments... Nov 5 18:52:36.162481568 (151h12m7.858330321s) [upload=133242]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 5.96296042s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [........X...X...............................................] duration: 6.207120322s trace: https://jaeger.dev.storj.io/trace/74cf1e30f6d49950 FAIL (2025-11-05 19:26:09 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.207152012s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [...........X..X.............................................] duration: 5.758528718s trace: https://jaeger.dev.storj.io/trace/3e2ae8884260b157 FAIL (2025-11-05 19:25:59 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.758557372s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [..........XX...X............................................] duration: 3.806149708s trace: https://jaeger.dev.storj.io/trace/204ab2570dc82d9c FAIL (2025-11-05 19:25:40 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.806181748s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [........X...X...............................................] duration: 6.202559605s trace: https://jaeger.dev.storj.io/trace/2e576aae48e8f03e FAIL (2025-11-05 19:25:34 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== 5 18:52:06.850167887 (151h11m38.54601663s) [noise=true , address=145.79.227.25:4618 , piece_id=XRUPY5BUPG7BXN2MI6MGYO4N3GW6TZDJE46MZFNRELWANHHG3N5A, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.850747301 (151h11m38.546596054s) [noise=true , address=77.90.31.15:4698 , piece_id=7VACL26N7ZCBXIA2I3XL5Q3UJ2BAHDIPXX4VZCQX3S63ABNW625Q, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.851222605 (151h11m38.547071358s) [noise=true , address=145.79.227.17:4972 , piece_id=H7UHQV7RDLCZUHY2XMXUAD2PX4ZBLWKTM47N6PH7UOGBRXPMWVOA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.851558575 (151h11m38.547407319s) [noise=true , address=145.79.227.22:4509 , piece_id=7RA5NJ72PI3DT4IC2ECVLZ3A2HXXL546ELVDXHZW4OXNN4D36OEA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.851909263 (151h11m38.547758006s) [noise=true , address=145.79.227.26:4654 , piece_id=E5J22KNPQSETSY443QQZXFFKUUJFHWFV22SQAVZ4BFQ4M6YWCNQQ, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.852327540 (151h11m38.548176293s) [noise=true , address=45.91.116.7:4010 , piece_id=EBVGGNQF3N4MTUIATJUWK4QZN7SEOWI6SIAW6N7JZOB5G6H33NEQ, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.852728624 (151h11m38.548577377s) [noise=true , address=145.79.227.9:4366 , piece_id=7ZRECKBQC4S6QLERBZY442WYS4H35XWOZYSGGPHSU7NQ6WL4UNVA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.853978774 (151h11m38.549827528s) [noise=true , address=77.90.31.18:4805 , piece_id=U2U34KK4AJK4A5HGNTHEYCIARLAR4X5NLOVCF63TN3PJUIRMMIHA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.854447296 (151h11m38.550296059s) [noise=true , address=77.90.31.16:4734 , piece_id=HFWVFM46PZFLZT6YNZVIA6HKLVN2PTV2GETZEKX356DMDGQO4HOA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.856523089 (151h11m38.552371842s) [noise=true , address=45.91.116.11:4117 , piece_id=QUGUYHEJWEEV733CBC3ONMV7XWDKVH2OFLQWXPQHPKZ7GTH4RLUA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.859490379 (151h11m38.555339122s) [noise=true , address=77.90.31.13:4628 , piece_id=QFTEHZ5FY7A5DPZZ44TRXGFRHPRNJZJHLQZDGRGOWYFEZYIZT2NQ, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.861350291 (151h11m38.557199034s) [noise=true , address=145.79.227.31:4797 , piece_id=2TFSZNURIAU573QUA5DUS7AJZNY2OXC6TJL3ST4VXIWY2WVB5YJA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.863368665 (151h11m38.559217418s) [noise=true , address=145.79.227.23:4547 , piece_id=K54H437KW5AAYUWOTMDBDP5GJ3ZABFKLAAOSS6WS5G5CROYEHODQ, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.864848794 (151h11m38.560697547s) [noise=true , address=145.79.227.24:4581 , piece_id=TSCVCH2KJEVMD4AVOEIHGTMRCRZ6NZMZA2CTQ6FUO2EIL3JAVYEA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.865644930 (151h11m38.561493683s) [noise=true , address=145.79.227.8:4330 , piece_id=4A47HIUX2MTJ2673XZZCMMF4T235TS4JWMNYJEVHHCKAHQX6APMA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:08.560504137 (151h11m40.25635289s) [upload=133242]: Next returned nil segment Nov 5 18:52:08.560522693 (151h11m40.256371446s) [upload=133242]: Waiting for error group managing segments... Nov 5 18:52:08.850333179 (151h11m40.546181932s) [noise=true , address=145.79.227.17:4972 , piece_id=H7UHQV7RDLCZUHY2XMXUAD2PX4ZBLWKTM47N6PH7UOGBRXPMWVOA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.853517822 (151h11m40.549366575s) [noise=true , address=145.79.227.22:4509 , piece_id=7RA5NJ72PI3DT4IC2ECVLZ3A2HXXL546ELVDXHZW4OXNN4D36OEA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.855547508 (151h11m40.551396261s) [noise=true , address=145.79.227.9:4366 , piece_id=7ZRECKBQC4S6QLERBZY442WYS4H35XWOZYSGGPHSU7NQ6WL4UNVA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.856103106 (151h11m40.551951859s) [noise=true , address=145.79.227.25:4618 , piece_id=XRUPY5BUPG7BXN2MI6MGYO4N3GW6TZDJE46MZFNRELWANHHG3N5A, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.860417321 (151h11m40.556266064s) [noise=true , address=145.79.227.8:4330 , piece_id=4A47HIUX2MTJ2673XZZCMMF4T235TS4JWMNYJEVHHCKAHQX6APMA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.865602796 (151h11m40.561451549s) [noise=true , address=145.79.227.23:4547 , piece_id=K54H437KW5AAYUWOTMDBDP5GJ3ZABFKLAAOSS6WS5G5CROYEHODQ, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.879393757 (151h11m40.57524251s) [noise=true , address=145.79.227.31:4797 , piece_id=2TFSZNURIAU573QUA5DUS7AJZNY2OXC6TJL3ST4VXIWY2WVB5YJA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.905541449 (151h11m40.601390202s) [noise=true , address=145.79.227.26:4654 , piece_id=E5J22KNPQSETSY443QQZXFFKUUJFHWFV22SQAVZ4BFQ4M6YWCNQQ, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.929018836 (151h11m40.624867589s) [noise=true , address=145.79.227.24:4581 , piece_id=TSCVCH2KJEVMD4AVOEIHGTMRCRZ6NZMZA2CTQ6FUO2EIL3JAVYEA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:35.899554917 (151h12m7.59540368s) [noise=true , address=77.90.31.17:4771 , piece_id=KVFE7VYW3RZJ25BWSH4BIRRQCMWDYOIK7EJ62XVREGN26YN7XYMQ, seg_pos={0 0}, upload=133242]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: write tcp 188.166.225.170:60676->77.90.31.17:4771: use of closed network connection; write tcp 188.166.225.170:60676->77.90.31.17:4771: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60676->77.90.31.17:4771: use of closed network connection Nov 5 18:52:35.899727506 (151h12m7.595576259s) [noise=true , address=77.90.31.13:4628 , piece_id=QFTEHZ5FY7A5DPZZ44TRXGFRHPRNJZJHLQZDGRGOWYFEZYIZT2NQ, seg_pos={0 0}, upload=133242]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: write tcp 188.166.225.170:42026->77.90.31.13:4628: use of closed network connection; write tcp 188.166.225.170:42026->77.90.31.13:4628: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42026->77.90.31.13:4628: use of closed network connection Nov 5 18:52:35.899860439 (151h12m7.595709192s) [noise=true , address=45.91.116.7:4010 , piece_id=EBVGGNQF3N4MTUIATJUWK4QZN7SEOWI6SIAW6N7JZOB5G6H33NEQ, seg_pos={0 0}, upload=133242]: Done uploading piece. err: ecclient: upload failed (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, address:45.91.116.7:4010): protocol: write tcp 188.166.225.170:40036->45.91.116.7:4010: use of closed network connection; write tcp 188.166.225.170:40036->45.91.116.7:4010: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40036->45.91.116.7:4010: use of closed network connection Nov 5 18:52:35.899923829 (151h12m7.595772572s) [noise=true , address=77.90.31.18:4805 , piece_id=U2U34KK4AJK4A5HGNTHEYCIARLAR4X5NLOVCF63TN3PJUIRMMIHA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: write tcp 188.166.225.170:60758->77.90.31.18:4805: use of closed network connection; write tcp 188.166.225.170:60758->77.90.31.18:4805: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60758->77.90.31.18:4805: use of closed network connection Nov 5 18:52:35.900052084 (151h12m7.595900837s) [noise=true , address=77.90.31.16:4734 , piece_id=HFWVFM46PZFLZT6YNZVIA6HKLVN2PTV2GETZEKX356DMDGQO4HOA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: write tcp 188.166.225.170:50728->77.90.31.16:4734: use of closed network connection; write tcp 188.166.225.170:50728->77.90.31.16:4734: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50728->77.90.31.16:4734: use of closed network connection Nov 5 18:52:35.900231175 (151h12m7.596079938s) [noise=true , address=45.91.116.11:4117 , piece_id=QUGUYHEJWEEV733CBC3ONMV7XWDKVH2OFLQWXPQHPKZ7GTH4RLUA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: ecclient: upload failed (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, address:45.91.116.11:4117): protocol: write tcp 188.166.225.170:41292->45.91.116.11:4117: use of closed network connection; write tcp 188.166.225.170:41292->45.91.116.11:4117: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41292->45.91.116.11:4117: use of closed network connection Nov 5 18:52:35.900342116 (151h12m7.596190859s) [noise=true , address=77.90.31.15:4698 , piece_id=7VACL26N7ZCBXIA2I3XL5Q3UJ2BAHDIPXX4VZCQX3S63ABNW625Q, seg_pos={0 0}, upload=133242]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: write tcp 188.166.225.170:41118->77.90.31.15:4698: use of closed network connection; write tcp 188.166.225.170:41118->77.90.31.15:4698: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41118->77.90.31.15:4698: use of closed network connection Nov 5 18:52:35.900382814 (151h12m7.596231567s) [seg_pos={0 0}, upload=133242]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Nov 5 18:52:36.162442033 (151h12m7.858290786s) [upload=133242]: Done uploading segments... Nov 5 18:52:36.162481568 (151h12m7.858330321s) [upload=133242]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 6.202578019s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.........X...X..............................................] duration: 6.19112697s trace: https://jaeger.dev.storj.io/trace/74cf1e30f6d49950 FAIL (2025-11-05 19:25:18 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.191244403s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [...........XX...X...........................................] duration: 3.538929514s trace: https://jaeger.dev.storj.io/trace/204ab2570dc82d9c FAIL (2025-11-05 19:25:14 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.539151777s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.........X...X..............................................] duration: 5.985720295s trace: https://jaeger.dev.storj.io/trace/2e576aae48e8f03e FAIL (2025-11-05 19:25:09 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== 5 18:52:06.850167887 (151h11m38.54601663s) [noise=true , address=145.79.227.25:4618 , piece_id=XRUPY5BUPG7BXN2MI6MGYO4N3GW6TZDJE46MZFNRELWANHHG3N5A, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.850747301 (151h11m38.546596054s) [noise=true , address=77.90.31.15:4698 , piece_id=7VACL26N7ZCBXIA2I3XL5Q3UJ2BAHDIPXX4VZCQX3S63ABNW625Q, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.851222605 (151h11m38.547071358s) [noise=true , address=145.79.227.17:4972 , piece_id=H7UHQV7RDLCZUHY2XMXUAD2PX4ZBLWKTM47N6PH7UOGBRXPMWVOA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.851558575 (151h11m38.547407319s) [noise=true , address=145.79.227.22:4509 , piece_id=7RA5NJ72PI3DT4IC2ECVLZ3A2HXXL546ELVDXHZW4OXNN4D36OEA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.851909263 (151h11m38.547758006s) [noise=true , address=145.79.227.26:4654 , piece_id=E5J22KNPQSETSY443QQZXFFKUUJFHWFV22SQAVZ4BFQ4M6YWCNQQ, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.852327540 (151h11m38.548176293s) [noise=true , address=45.91.116.7:4010 , piece_id=EBVGGNQF3N4MTUIATJUWK4QZN7SEOWI6SIAW6N7JZOB5G6H33NEQ, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.852728624 (151h11m38.548577377s) [noise=true , address=145.79.227.9:4366 , piece_id=7ZRECKBQC4S6QLERBZY442WYS4H35XWOZYSGGPHSU7NQ6WL4UNVA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.853978774 (151h11m38.549827528s) [noise=true , address=77.90.31.18:4805 , piece_id=U2U34KK4AJK4A5HGNTHEYCIARLAR4X5NLOVCF63TN3PJUIRMMIHA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.854447296 (151h11m38.550296059s) [noise=true , address=77.90.31.16:4734 , piece_id=HFWVFM46PZFLZT6YNZVIA6HKLVN2PTV2GETZEKX356DMDGQO4HOA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.856523089 (151h11m38.552371842s) [noise=true , address=45.91.116.11:4117 , piece_id=QUGUYHEJWEEV733CBC3ONMV7XWDKVH2OFLQWXPQHPKZ7GTH4RLUA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.859490379 (151h11m38.555339122s) [noise=true , address=77.90.31.13:4628 , piece_id=QFTEHZ5FY7A5DPZZ44TRXGFRHPRNJZJHLQZDGRGOWYFEZYIZT2NQ, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.861350291 (151h11m38.557199034s) [noise=true , address=145.79.227.31:4797 , piece_id=2TFSZNURIAU573QUA5DUS7AJZNY2OXC6TJL3ST4VXIWY2WVB5YJA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.863368665 (151h11m38.559217418s) [noise=true , address=145.79.227.23:4547 , piece_id=K54H437KW5AAYUWOTMDBDP5GJ3ZABFKLAAOSS6WS5G5CROYEHODQ, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.864848794 (151h11m38.560697547s) [noise=true , address=145.79.227.24:4581 , piece_id=TSCVCH2KJEVMD4AVOEIHGTMRCRZ6NZMZA2CTQ6FUO2EIL3JAVYEA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.865644930 (151h11m38.561493683s) [noise=true , address=145.79.227.8:4330 , piece_id=4A47HIUX2MTJ2673XZZCMMF4T235TS4JWMNYJEVHHCKAHQX6APMA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:08.560504137 (151h11m40.25635289s) [upload=133242]: Next returned nil segment Nov 5 18:52:08.560522693 (151h11m40.256371446s) [upload=133242]: Waiting for error group managing segments... Nov 5 18:52:08.850333179 (151h11m40.546181932s) [noise=true , address=145.79.227.17:4972 , piece_id=H7UHQV7RDLCZUHY2XMXUAD2PX4ZBLWKTM47N6PH7UOGBRXPMWVOA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.853517822 (151h11m40.549366575s) [noise=true , address=145.79.227.22:4509 , piece_id=7RA5NJ72PI3DT4IC2ECVLZ3A2HXXL546ELVDXHZW4OXNN4D36OEA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.855547508 (151h11m40.551396261s) [noise=true , address=145.79.227.9:4366 , piece_id=7ZRECKBQC4S6QLERBZY442WYS4H35XWOZYSGGPHSU7NQ6WL4UNVA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.856103106 (151h11m40.551951859s) [noise=true , address=145.79.227.25:4618 , piece_id=XRUPY5BUPG7BXN2MI6MGYO4N3GW6TZDJE46MZFNRELWANHHG3N5A, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.860417321 (151h11m40.556266064s) [noise=true , address=145.79.227.8:4330 , piece_id=4A47HIUX2MTJ2673XZZCMMF4T235TS4JWMNYJEVHHCKAHQX6APMA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.865602796 (151h11m40.561451549s) [noise=true , address=145.79.227.23:4547 , piece_id=K54H437KW5AAYUWOTMDBDP5GJ3ZABFKLAAOSS6WS5G5CROYEHODQ, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.879393757 (151h11m40.57524251s) [noise=true , address=145.79.227.31:4797 , piece_id=2TFSZNURIAU573QUA5DUS7AJZNY2OXC6TJL3ST4VXIWY2WVB5YJA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.905541449 (151h11m40.601390202s) [noise=true , address=145.79.227.26:4654 , piece_id=E5J22KNPQSETSY443QQZXFFKUUJFHWFV22SQAVZ4BFQ4M6YWCNQQ, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.929018836 (151h11m40.624867589s) [noise=true , address=145.79.227.24:4581 , piece_id=TSCVCH2KJEVMD4AVOEIHGTMRCRZ6NZMZA2CTQ6FUO2EIL3JAVYEA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:35.899554917 (151h12m7.59540368s) [noise=true , address=77.90.31.17:4771 , piece_id=KVFE7VYW3RZJ25BWSH4BIRRQCMWDYOIK7EJ62XVREGN26YN7XYMQ, seg_pos={0 0}, upload=133242]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: write tcp 188.166.225.170:60676->77.90.31.17:4771: use of closed network connection; write tcp 188.166.225.170:60676->77.90.31.17:4771: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60676->77.90.31.17:4771: use of closed network connection Nov 5 18:52:35.899727506 (151h12m7.595576259s) [noise=true , address=77.90.31.13:4628 , piece_id=QFTEHZ5FY7A5DPZZ44TRXGFRHPRNJZJHLQZDGRGOWYFEZYIZT2NQ, seg_pos={0 0}, upload=133242]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: write tcp 188.166.225.170:42026->77.90.31.13:4628: use of closed network connection; write tcp 188.166.225.170:42026->77.90.31.13:4628: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42026->77.90.31.13:4628: use of closed network connection Nov 5 18:52:35.899860439 (151h12m7.595709192s) [noise=true , address=45.91.116.7:4010 , piece_id=EBVGGNQF3N4MTUIATJUWK4QZN7SEOWI6SIAW6N7JZOB5G6H33NEQ, seg_pos={0 0}, upload=133242]: Done uploading piece. err: ecclient: upload failed (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, address:45.91.116.7:4010): protocol: write tcp 188.166.225.170:40036->45.91.116.7:4010: use of closed network connection; write tcp 188.166.225.170:40036->45.91.116.7:4010: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40036->45.91.116.7:4010: use of closed network connection Nov 5 18:52:35.899923829 (151h12m7.595772572s) [noise=true , address=77.90.31.18:4805 , piece_id=U2U34KK4AJK4A5HGNTHEYCIARLAR4X5NLOVCF63TN3PJUIRMMIHA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: write tcp 188.166.225.170:60758->77.90.31.18:4805: use of closed network connection; write tcp 188.166.225.170:60758->77.90.31.18:4805: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60758->77.90.31.18:4805: use of closed network connection Nov 5 18:52:35.900052084 (151h12m7.595900837s) [noise=true , address=77.90.31.16:4734 , piece_id=HFWVFM46PZFLZT6YNZVIA6HKLVN2PTV2GETZEKX356DMDGQO4HOA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: write tcp 188.166.225.170:50728->77.90.31.16:4734: use of closed network connection; write tcp 188.166.225.170:50728->77.90.31.16:4734: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50728->77.90.31.16:4734: use of closed network connection Nov 5 18:52:35.900231175 (151h12m7.596079938s) [noise=true , address=45.91.116.11:4117 , piece_id=QUGUYHEJWEEV733CBC3ONMV7XWDKVH2OFLQWXPQHPKZ7GTH4RLUA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: ecclient: upload failed (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, address:45.91.116.11:4117): protocol: write tcp 188.166.225.170:41292->45.91.116.11:4117: use of closed network connection; write tcp 188.166.225.170:41292->45.91.116.11:4117: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41292->45.91.116.11:4117: use of closed network connection Nov 5 18:52:35.900342116 (151h12m7.596190859s) [noise=true , address=77.90.31.15:4698 , piece_id=7VACL26N7ZCBXIA2I3XL5Q3UJ2BAHDIPXX4VZCQX3S63ABNW625Q, seg_pos={0 0}, upload=133242]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: write tcp 188.166.225.170:41118->77.90.31.15:4698: use of closed network connection; write tcp 188.166.225.170:41118->77.90.31.15:4698: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41118->77.90.31.15:4698: use of closed network connection Nov 5 18:52:35.900382814 (151h12m7.596231567s) [seg_pos={0 0}, upload=133242]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Nov 5 18:52:36.162442033 (151h12m7.858290786s) [upload=133242]: Done uploading segments... Nov 5 18:52:36.162481568 (151h12m7.858330321s) [upload=133242]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 5.985752627s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [............X..X............................................] duration: 5.742312366s trace: https://jaeger.dev.storj.io/trace/3e2ae8884260b157 FAIL (2025-11-05 19:24:51 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.742331352s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..........X...X.............................................] duration: 6.412621431s trace: https://jaeger.dev.storj.io/trace/74cf1e30f6d49950 FAIL (2025-11-05 19:24:35 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.412670815s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...........X...X............................................] duration: 6.476409739s trace: https://jaeger.dev.storj.io/trace/74cf1e30f6d49950 FAIL (2025-11-05 19:24:10 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.476440085s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [............XX...X..........................................] duration: 3.396688647s trace: https://jaeger.dev.storj.io/trace/204ab2570dc82d9c FAIL (2025-11-05 19:24:06 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.396724845s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..........X...X.............................................] duration: 6.017041756s trace: https://jaeger.dev.storj.io/trace/2e576aae48e8f03e FAIL (2025-11-05 19:23:59 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== 5 18:52:06.850167887 (151h11m38.54601663s) [noise=true , address=145.79.227.25:4618 , piece_id=XRUPY5BUPG7BXN2MI6MGYO4N3GW6TZDJE46MZFNRELWANHHG3N5A, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.850747301 (151h11m38.546596054s) [noise=true , address=77.90.31.15:4698 , piece_id=7VACL26N7ZCBXIA2I3XL5Q3UJ2BAHDIPXX4VZCQX3S63ABNW625Q, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.851222605 (151h11m38.547071358s) [noise=true , address=145.79.227.17:4972 , piece_id=H7UHQV7RDLCZUHY2XMXUAD2PX4ZBLWKTM47N6PH7UOGBRXPMWVOA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.851558575 (151h11m38.547407319s) [noise=true , address=145.79.227.22:4509 , piece_id=7RA5NJ72PI3DT4IC2ECVLZ3A2HXXL546ELVDXHZW4OXNN4D36OEA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.851909263 (151h11m38.547758006s) [noise=true , address=145.79.227.26:4654 , piece_id=E5J22KNPQSETSY443QQZXFFKUUJFHWFV22SQAVZ4BFQ4M6YWCNQQ, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.852327540 (151h11m38.548176293s) [noise=true , address=45.91.116.7:4010 , piece_id=EBVGGNQF3N4MTUIATJUWK4QZN7SEOWI6SIAW6N7JZOB5G6H33NEQ, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.852728624 (151h11m38.548577377s) [noise=true , address=145.79.227.9:4366 , piece_id=7ZRECKBQC4S6QLERBZY442WYS4H35XWOZYSGGPHSU7NQ6WL4UNVA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.853978774 (151h11m38.549827528s) [noise=true , address=77.90.31.18:4805 , piece_id=U2U34KK4AJK4A5HGNTHEYCIARLAR4X5NLOVCF63TN3PJUIRMMIHA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.854447296 (151h11m38.550296059s) [noise=true , address=77.90.31.16:4734 , piece_id=HFWVFM46PZFLZT6YNZVIA6HKLVN2PTV2GETZEKX356DMDGQO4HOA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.856523089 (151h11m38.552371842s) [noise=true , address=45.91.116.11:4117 , piece_id=QUGUYHEJWEEV733CBC3ONMV7XWDKVH2OFLQWXPQHPKZ7GTH4RLUA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.859490379 (151h11m38.555339122s) [noise=true , address=77.90.31.13:4628 , piece_id=QFTEHZ5FY7A5DPZZ44TRXGFRHPRNJZJHLQZDGRGOWYFEZYIZT2NQ, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.861350291 (151h11m38.557199034s) [noise=true , address=145.79.227.31:4797 , piece_id=2TFSZNURIAU573QUA5DUS7AJZNY2OXC6TJL3ST4VXIWY2WVB5YJA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.863368665 (151h11m38.559217418s) [noise=true , address=145.79.227.23:4547 , piece_id=K54H437KW5AAYUWOTMDBDP5GJ3ZABFKLAAOSS6WS5G5CROYEHODQ, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.864848794 (151h11m38.560697547s) [noise=true , address=145.79.227.24:4581 , piece_id=TSCVCH2KJEVMD4AVOEIHGTMRCRZ6NZMZA2CTQ6FUO2EIL3JAVYEA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.865644930 (151h11m38.561493683s) [noise=true , address=145.79.227.8:4330 , piece_id=4A47HIUX2MTJ2673XZZCMMF4T235TS4JWMNYJEVHHCKAHQX6APMA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:08.560504137 (151h11m40.25635289s) [upload=133242]: Next returned nil segment Nov 5 18:52:08.560522693 (151h11m40.256371446s) [upload=133242]: Waiting for error group managing segments... Nov 5 18:52:08.850333179 (151h11m40.546181932s) [noise=true , address=145.79.227.17:4972 , piece_id=H7UHQV7RDLCZUHY2XMXUAD2PX4ZBLWKTM47N6PH7UOGBRXPMWVOA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.853517822 (151h11m40.549366575s) [noise=true , address=145.79.227.22:4509 , piece_id=7RA5NJ72PI3DT4IC2ECVLZ3A2HXXL546ELVDXHZW4OXNN4D36OEA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.855547508 (151h11m40.551396261s) [noise=true , address=145.79.227.9:4366 , piece_id=7ZRECKBQC4S6QLERBZY442WYS4H35XWOZYSGGPHSU7NQ6WL4UNVA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.856103106 (151h11m40.551951859s) [noise=true , address=145.79.227.25:4618 , piece_id=XRUPY5BUPG7BXN2MI6MGYO4N3GW6TZDJE46MZFNRELWANHHG3N5A, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.860417321 (151h11m40.556266064s) [noise=true , address=145.79.227.8:4330 , piece_id=4A47HIUX2MTJ2673XZZCMMF4T235TS4JWMNYJEVHHCKAHQX6APMA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.865602796 (151h11m40.561451549s) [noise=true , address=145.79.227.23:4547 , piece_id=K54H437KW5AAYUWOTMDBDP5GJ3ZABFKLAAOSS6WS5G5CROYEHODQ, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.879393757 (151h11m40.57524251s) [noise=true , address=145.79.227.31:4797 , piece_id=2TFSZNURIAU573QUA5DUS7AJZNY2OXC6TJL3ST4VXIWY2WVB5YJA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.905541449 (151h11m40.601390202s) [noise=true , address=145.79.227.26:4654 , piece_id=E5J22KNPQSETSY443QQZXFFKUUJFHWFV22SQAVZ4BFQ4M6YWCNQQ, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.929018836 (151h11m40.624867589s) [noise=true , address=145.79.227.24:4581 , piece_id=TSCVCH2KJEVMD4AVOEIHGTMRCRZ6NZMZA2CTQ6FUO2EIL3JAVYEA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:35.899554917 (151h12m7.59540368s) [noise=true , address=77.90.31.17:4771 , piece_id=KVFE7VYW3RZJ25BWSH4BIRRQCMWDYOIK7EJ62XVREGN26YN7XYMQ, seg_pos={0 0}, upload=133242]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: write tcp 188.166.225.170:60676->77.90.31.17:4771: use of closed network connection; write tcp 188.166.225.170:60676->77.90.31.17:4771: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60676->77.90.31.17:4771: use of closed network connection Nov 5 18:52:35.899727506 (151h12m7.595576259s) [noise=true , address=77.90.31.13:4628 , piece_id=QFTEHZ5FY7A5DPZZ44TRXGFRHPRNJZJHLQZDGRGOWYFEZYIZT2NQ, seg_pos={0 0}, upload=133242]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: write tcp 188.166.225.170:42026->77.90.31.13:4628: use of closed network connection; write tcp 188.166.225.170:42026->77.90.31.13:4628: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42026->77.90.31.13:4628: use of closed network connection Nov 5 18:52:35.899860439 (151h12m7.595709192s) [noise=true , address=45.91.116.7:4010 , piece_id=EBVGGNQF3N4MTUIATJUWK4QZN7SEOWI6SIAW6N7JZOB5G6H33NEQ, seg_pos={0 0}, upload=133242]: Done uploading piece. err: ecclient: upload failed (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, address:45.91.116.7:4010): protocol: write tcp 188.166.225.170:40036->45.91.116.7:4010: use of closed network connection; write tcp 188.166.225.170:40036->45.91.116.7:4010: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40036->45.91.116.7:4010: use of closed network connection Nov 5 18:52:35.899923829 (151h12m7.595772572s) [noise=true , address=77.90.31.18:4805 , piece_id=U2U34KK4AJK4A5HGNTHEYCIARLAR4X5NLOVCF63TN3PJUIRMMIHA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: write tcp 188.166.225.170:60758->77.90.31.18:4805: use of closed network connection; write tcp 188.166.225.170:60758->77.90.31.18:4805: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60758->77.90.31.18:4805: use of closed network connection Nov 5 18:52:35.900052084 (151h12m7.595900837s) [noise=true , address=77.90.31.16:4734 , piece_id=HFWVFM46PZFLZT6YNZVIA6HKLVN2PTV2GETZEKX356DMDGQO4HOA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: write tcp 188.166.225.170:50728->77.90.31.16:4734: use of closed network connection; write tcp 188.166.225.170:50728->77.90.31.16:4734: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50728->77.90.31.16:4734: use of closed network connection Nov 5 18:52:35.900231175 (151h12m7.596079938s) [noise=true , address=45.91.116.11:4117 , piece_id=QUGUYHEJWEEV733CBC3ONMV7XWDKVH2OFLQWXPQHPKZ7GTH4RLUA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: ecclient: upload failed (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, address:45.91.116.11:4117): protocol: write tcp 188.166.225.170:41292->45.91.116.11:4117: use of closed network connection; write tcp 188.166.225.170:41292->45.91.116.11:4117: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41292->45.91.116.11:4117: use of closed network connection Nov 5 18:52:35.900342116 (151h12m7.596190859s) [noise=true , address=77.90.31.15:4698 , piece_id=7VACL26N7ZCBXIA2I3XL5Q3UJ2BAHDIPXX4VZCQX3S63ABNW625Q, seg_pos={0 0}, upload=133242]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: write tcp 188.166.225.170:41118->77.90.31.15:4698: use of closed network connection; write tcp 188.166.225.170:41118->77.90.31.15:4698: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41118->77.90.31.15:4698: use of closed network connection Nov 5 18:52:35.900382814 (151h12m7.596231567s) [seg_pos={0 0}, upload=133242]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Nov 5 18:52:36.162442033 (151h12m7.858290786s) [upload=133242]: Done uploading segments... Nov 5 18:52:36.162481568 (151h12m7.858330321s) [upload=133242]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 6.017069369s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [............X...X...........................................] duration: 6.148640243s trace: https://jaeger.dev.storj.io/trace/74cf1e30f6d49950 FAIL (2025-11-05 19:23:51 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.148670542s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.............X..X...........................................] duration: 5.730683495s trace: https://jaeger.dev.storj.io/trace/3e2ae8884260b157 FAIL (2025-11-05 19:23:39 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.730711056s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [.............XX...X.........................................] duration: 3.546671101s trace: https://jaeger.dev.storj.io/trace/204ab2570dc82d9c FAIL (2025-11-05 19:23:22 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.546700496s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.............X...X..........................................] duration: 6.212507745s trace: https://jaeger.dev.storj.io/trace/74cf1e30f6d49950 FAIL (2025-11-05 19:23:17 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.212540388s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...........X...X............................................] duration: 5.778875969s trace: https://jaeger.dev.storj.io/trace/2e576aae48e8f03e FAIL (2025-11-05 19:22:54 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== 5 18:52:06.850167887 (151h11m38.54601663s) [noise=true , address=145.79.227.25:4618 , piece_id=XRUPY5BUPG7BXN2MI6MGYO4N3GW6TZDJE46MZFNRELWANHHG3N5A, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.850747301 (151h11m38.546596054s) [noise=true , address=77.90.31.15:4698 , piece_id=7VACL26N7ZCBXIA2I3XL5Q3UJ2BAHDIPXX4VZCQX3S63ABNW625Q, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.851222605 (151h11m38.547071358s) [noise=true , address=145.79.227.17:4972 , piece_id=H7UHQV7RDLCZUHY2XMXUAD2PX4ZBLWKTM47N6PH7UOGBRXPMWVOA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.851558575 (151h11m38.547407319s) [noise=true , address=145.79.227.22:4509 , piece_id=7RA5NJ72PI3DT4IC2ECVLZ3A2HXXL546ELVDXHZW4OXNN4D36OEA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.851909263 (151h11m38.547758006s) [noise=true , address=145.79.227.26:4654 , piece_id=E5J22KNPQSETSY443QQZXFFKUUJFHWFV22SQAVZ4BFQ4M6YWCNQQ, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.852327540 (151h11m38.548176293s) [noise=true , address=45.91.116.7:4010 , piece_id=EBVGGNQF3N4MTUIATJUWK4QZN7SEOWI6SIAW6N7JZOB5G6H33NEQ, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.852728624 (151h11m38.548577377s) [noise=true , address=145.79.227.9:4366 , piece_id=7ZRECKBQC4S6QLERBZY442WYS4H35XWOZYSGGPHSU7NQ6WL4UNVA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.853978774 (151h11m38.549827528s) [noise=true , address=77.90.31.18:4805 , piece_id=U2U34KK4AJK4A5HGNTHEYCIARLAR4X5NLOVCF63TN3PJUIRMMIHA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.854447296 (151h11m38.550296059s) [noise=true , address=77.90.31.16:4734 , piece_id=HFWVFM46PZFLZT6YNZVIA6HKLVN2PTV2GETZEKX356DMDGQO4HOA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.856523089 (151h11m38.552371842s) [noise=true , address=45.91.116.11:4117 , piece_id=QUGUYHEJWEEV733CBC3ONMV7XWDKVH2OFLQWXPQHPKZ7GTH4RLUA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.859490379 (151h11m38.555339122s) [noise=true , address=77.90.31.13:4628 , piece_id=QFTEHZ5FY7A5DPZZ44TRXGFRHPRNJZJHLQZDGRGOWYFEZYIZT2NQ, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.861350291 (151h11m38.557199034s) [noise=true , address=145.79.227.31:4797 , piece_id=2TFSZNURIAU573QUA5DUS7AJZNY2OXC6TJL3ST4VXIWY2WVB5YJA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.863368665 (151h11m38.559217418s) [noise=true , address=145.79.227.23:4547 , piece_id=K54H437KW5AAYUWOTMDBDP5GJ3ZABFKLAAOSS6WS5G5CROYEHODQ, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.864848794 (151h11m38.560697547s) [noise=true , address=145.79.227.24:4581 , piece_id=TSCVCH2KJEVMD4AVOEIHGTMRCRZ6NZMZA2CTQ6FUO2EIL3JAVYEA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.865644930 (151h11m38.561493683s) [noise=true , address=145.79.227.8:4330 , piece_id=4A47HIUX2MTJ2673XZZCMMF4T235TS4JWMNYJEVHHCKAHQX6APMA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:08.560504137 (151h11m40.25635289s) [upload=133242]: Next returned nil segment Nov 5 18:52:08.560522693 (151h11m40.256371446s) [upload=133242]: Waiting for error group managing segments... Nov 5 18:52:08.850333179 (151h11m40.546181932s) [noise=true , address=145.79.227.17:4972 , piece_id=H7UHQV7RDLCZUHY2XMXUAD2PX4ZBLWKTM47N6PH7UOGBRXPMWVOA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.853517822 (151h11m40.549366575s) [noise=true , address=145.79.227.22:4509 , piece_id=7RA5NJ72PI3DT4IC2ECVLZ3A2HXXL546ELVDXHZW4OXNN4D36OEA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.855547508 (151h11m40.551396261s) [noise=true , address=145.79.227.9:4366 , piece_id=7ZRECKBQC4S6QLERBZY442WYS4H35XWOZYSGGPHSU7NQ6WL4UNVA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.856103106 (151h11m40.551951859s) [noise=true , address=145.79.227.25:4618 , piece_id=XRUPY5BUPG7BXN2MI6MGYO4N3GW6TZDJE46MZFNRELWANHHG3N5A, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.860417321 (151h11m40.556266064s) [noise=true , address=145.79.227.8:4330 , piece_id=4A47HIUX2MTJ2673XZZCMMF4T235TS4JWMNYJEVHHCKAHQX6APMA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.865602796 (151h11m40.561451549s) [noise=true , address=145.79.227.23:4547 , piece_id=K54H437KW5AAYUWOTMDBDP5GJ3ZABFKLAAOSS6WS5G5CROYEHODQ, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.879393757 (151h11m40.57524251s) [noise=true , address=145.79.227.31:4797 , piece_id=2TFSZNURIAU573QUA5DUS7AJZNY2OXC6TJL3ST4VXIWY2WVB5YJA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.905541449 (151h11m40.601390202s) [noise=true , address=145.79.227.26:4654 , piece_id=E5J22KNPQSETSY443QQZXFFKUUJFHWFV22SQAVZ4BFQ4M6YWCNQQ, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.929018836 (151h11m40.624867589s) [noise=true , address=145.79.227.24:4581 , piece_id=TSCVCH2KJEVMD4AVOEIHGTMRCRZ6NZMZA2CTQ6FUO2EIL3JAVYEA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:35.899554917 (151h12m7.59540368s) [noise=true , address=77.90.31.17:4771 , piece_id=KVFE7VYW3RZJ25BWSH4BIRRQCMWDYOIK7EJ62XVREGN26YN7XYMQ, seg_pos={0 0}, upload=133242]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: write tcp 188.166.225.170:60676->77.90.31.17:4771: use of closed network connection; write tcp 188.166.225.170:60676->77.90.31.17:4771: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60676->77.90.31.17:4771: use of closed network connection Nov 5 18:52:35.899727506 (151h12m7.595576259s) [noise=true , address=77.90.31.13:4628 , piece_id=QFTEHZ5FY7A5DPZZ44TRXGFRHPRNJZJHLQZDGRGOWYFEZYIZT2NQ, seg_pos={0 0}, upload=133242]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: write tcp 188.166.225.170:42026->77.90.31.13:4628: use of closed network connection; write tcp 188.166.225.170:42026->77.90.31.13:4628: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42026->77.90.31.13:4628: use of closed network connection Nov 5 18:52:35.899860439 (151h12m7.595709192s) [noise=true , address=45.91.116.7:4010 , piece_id=EBVGGNQF3N4MTUIATJUWK4QZN7SEOWI6SIAW6N7JZOB5G6H33NEQ, seg_pos={0 0}, upload=133242]: Done uploading piece. err: ecclient: upload failed (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, address:45.91.116.7:4010): protocol: write tcp 188.166.225.170:40036->45.91.116.7:4010: use of closed network connection; write tcp 188.166.225.170:40036->45.91.116.7:4010: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40036->45.91.116.7:4010: use of closed network connection Nov 5 18:52:35.899923829 (151h12m7.595772572s) [noise=true , address=77.90.31.18:4805 , piece_id=U2U34KK4AJK4A5HGNTHEYCIARLAR4X5NLOVCF63TN3PJUIRMMIHA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: write tcp 188.166.225.170:60758->77.90.31.18:4805: use of closed network connection; write tcp 188.166.225.170:60758->77.90.31.18:4805: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60758->77.90.31.18:4805: use of closed network connection Nov 5 18:52:35.900052084 (151h12m7.595900837s) [noise=true , address=77.90.31.16:4734 , piece_id=HFWVFM46PZFLZT6YNZVIA6HKLVN2PTV2GETZEKX356DMDGQO4HOA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: write tcp 188.166.225.170:50728->77.90.31.16:4734: use of closed network connection; write tcp 188.166.225.170:50728->77.90.31.16:4734: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50728->77.90.31.16:4734: use of closed network connection Nov 5 18:52:35.900231175 (151h12m7.596079938s) [noise=true , address=45.91.116.11:4117 , piece_id=QUGUYHEJWEEV733CBC3ONMV7XWDKVH2OFLQWXPQHPKZ7GTH4RLUA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: ecclient: upload failed (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, address:45.91.116.11:4117): protocol: write tcp 188.166.225.170:41292->45.91.116.11:4117: use of closed network connection; write tcp 188.166.225.170:41292->45.91.116.11:4117: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41292->45.91.116.11:4117: use of closed network connection Nov 5 18:52:35.900342116 (151h12m7.596190859s) [noise=true , address=77.90.31.15:4698 , piece_id=7VACL26N7ZCBXIA2I3XL5Q3UJ2BAHDIPXX4VZCQX3S63ABNW625Q, seg_pos={0 0}, upload=133242]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: write tcp 188.166.225.170:41118->77.90.31.15:4698: use of closed network connection; write tcp 188.166.225.170:41118->77.90.31.15:4698: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41118->77.90.31.15:4698: use of closed network connection Nov 5 18:52:35.900382814 (151h12m7.596231567s) [seg_pos={0 0}, upload=133242]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Nov 5 18:52:36.162442033 (151h12m7.858290786s) [upload=133242]: Done uploading segments... Nov 5 18:52:36.162481568 (151h12m7.858330321s) [upload=133242]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 5.778987953s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [..............XX...X........................................] duration: 3.442515664s trace: https://jaeger.dev.storj.io/trace/204ab2570dc82d9c FAIL (2025-11-05 19:22:52 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.442547495s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [............X...X...........................................] duration: 5.89790587s trace: https://jaeger.dev.storj.io/trace/2e576aae48e8f03e FAIL (2025-11-05 19:22:35 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== 5 18:52:06.850167887 (151h11m38.54601663s) [noise=true , address=145.79.227.25:4618 , piece_id=XRUPY5BUPG7BXN2MI6MGYO4N3GW6TZDJE46MZFNRELWANHHG3N5A, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.850747301 (151h11m38.546596054s) [noise=true , address=77.90.31.15:4698 , piece_id=7VACL26N7ZCBXIA2I3XL5Q3UJ2BAHDIPXX4VZCQX3S63ABNW625Q, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.851222605 (151h11m38.547071358s) [noise=true , address=145.79.227.17:4972 , piece_id=H7UHQV7RDLCZUHY2XMXUAD2PX4ZBLWKTM47N6PH7UOGBRXPMWVOA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.851558575 (151h11m38.547407319s) [noise=true , address=145.79.227.22:4509 , piece_id=7RA5NJ72PI3DT4IC2ECVLZ3A2HXXL546ELVDXHZW4OXNN4D36OEA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.851909263 (151h11m38.547758006s) [noise=true , address=145.79.227.26:4654 , piece_id=E5J22KNPQSETSY443QQZXFFKUUJFHWFV22SQAVZ4BFQ4M6YWCNQQ, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.852327540 (151h11m38.548176293s) [noise=true , address=45.91.116.7:4010 , piece_id=EBVGGNQF3N4MTUIATJUWK4QZN7SEOWI6SIAW6N7JZOB5G6H33NEQ, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.852728624 (151h11m38.548577377s) [noise=true , address=145.79.227.9:4366 , piece_id=7ZRECKBQC4S6QLERBZY442WYS4H35XWOZYSGGPHSU7NQ6WL4UNVA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.853978774 (151h11m38.549827528s) [noise=true , address=77.90.31.18:4805 , piece_id=U2U34KK4AJK4A5HGNTHEYCIARLAR4X5NLOVCF63TN3PJUIRMMIHA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.854447296 (151h11m38.550296059s) [noise=true , address=77.90.31.16:4734 , piece_id=HFWVFM46PZFLZT6YNZVIA6HKLVN2PTV2GETZEKX356DMDGQO4HOA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.856523089 (151h11m38.552371842s) [noise=true , address=45.91.116.11:4117 , piece_id=QUGUYHEJWEEV733CBC3ONMV7XWDKVH2OFLQWXPQHPKZ7GTH4RLUA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.859490379 (151h11m38.555339122s) [noise=true , address=77.90.31.13:4628 , piece_id=QFTEHZ5FY7A5DPZZ44TRXGFRHPRNJZJHLQZDGRGOWYFEZYIZT2NQ, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.861350291 (151h11m38.557199034s) [noise=true , address=145.79.227.31:4797 , piece_id=2TFSZNURIAU573QUA5DUS7AJZNY2OXC6TJL3ST4VXIWY2WVB5YJA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.863368665 (151h11m38.559217418s) [noise=true , address=145.79.227.23:4547 , piece_id=K54H437KW5AAYUWOTMDBDP5GJ3ZABFKLAAOSS6WS5G5CROYEHODQ, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.864848794 (151h11m38.560697547s) [noise=true , address=145.79.227.24:4581 , piece_id=TSCVCH2KJEVMD4AVOEIHGTMRCRZ6NZMZA2CTQ6FUO2EIL3JAVYEA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.865644930 (151h11m38.561493683s) [noise=true , address=145.79.227.8:4330 , piece_id=4A47HIUX2MTJ2673XZZCMMF4T235TS4JWMNYJEVHHCKAHQX6APMA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:08.560504137 (151h11m40.25635289s) [upload=133242]: Next returned nil segment Nov 5 18:52:08.560522693 (151h11m40.256371446s) [upload=133242]: Waiting for error group managing segments... Nov 5 18:52:08.850333179 (151h11m40.546181932s) [noise=true , address=145.79.227.17:4972 , piece_id=H7UHQV7RDLCZUHY2XMXUAD2PX4ZBLWKTM47N6PH7UOGBRXPMWVOA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.853517822 (151h11m40.549366575s) [noise=true , address=145.79.227.22:4509 , piece_id=7RA5NJ72PI3DT4IC2ECVLZ3A2HXXL546ELVDXHZW4OXNN4D36OEA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.855547508 (151h11m40.551396261s) [noise=true , address=145.79.227.9:4366 , piece_id=7ZRECKBQC4S6QLERBZY442WYS4H35XWOZYSGGPHSU7NQ6WL4UNVA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.856103106 (151h11m40.551951859s) [noise=true , address=145.79.227.25:4618 , piece_id=XRUPY5BUPG7BXN2MI6MGYO4N3GW6TZDJE46MZFNRELWANHHG3N5A, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.860417321 (151h11m40.556266064s) [noise=true , address=145.79.227.8:4330 , piece_id=4A47HIUX2MTJ2673XZZCMMF4T235TS4JWMNYJEVHHCKAHQX6APMA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.865602796 (151h11m40.561451549s) [noise=true , address=145.79.227.23:4547 , piece_id=K54H437KW5AAYUWOTMDBDP5GJ3ZABFKLAAOSS6WS5G5CROYEHODQ, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.879393757 (151h11m40.57524251s) [noise=true , address=145.79.227.31:4797 , piece_id=2TFSZNURIAU573QUA5DUS7AJZNY2OXC6TJL3ST4VXIWY2WVB5YJA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.905541449 (151h11m40.601390202s) [noise=true , address=145.79.227.26:4654 , piece_id=E5J22KNPQSETSY443QQZXFFKUUJFHWFV22SQAVZ4BFQ4M6YWCNQQ, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.929018836 (151h11m40.624867589s) [noise=true , address=145.79.227.24:4581 , piece_id=TSCVCH2KJEVMD4AVOEIHGTMRCRZ6NZMZA2CTQ6FUO2EIL3JAVYEA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:35.899554917 (151h12m7.59540368s) [noise=true , address=77.90.31.17:4771 , piece_id=KVFE7VYW3RZJ25BWSH4BIRRQCMWDYOIK7EJ62XVREGN26YN7XYMQ, seg_pos={0 0}, upload=133242]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: write tcp 188.166.225.170:60676->77.90.31.17:4771: use of closed network connection; write tcp 188.166.225.170:60676->77.90.31.17:4771: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60676->77.90.31.17:4771: use of closed network connection Nov 5 18:52:35.899727506 (151h12m7.595576259s) [noise=true , address=77.90.31.13:4628 , piece_id=QFTEHZ5FY7A5DPZZ44TRXGFRHPRNJZJHLQZDGRGOWYFEZYIZT2NQ, seg_pos={0 0}, upload=133242]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: write tcp 188.166.225.170:42026->77.90.31.13:4628: use of closed network connection; write tcp 188.166.225.170:42026->77.90.31.13:4628: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42026->77.90.31.13:4628: use of closed network connection Nov 5 18:52:35.899860439 (151h12m7.595709192s) [noise=true , address=45.91.116.7:4010 , piece_id=EBVGGNQF3N4MTUIATJUWK4QZN7SEOWI6SIAW6N7JZOB5G6H33NEQ, seg_pos={0 0}, upload=133242]: Done uploading piece. err: ecclient: upload failed (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, address:45.91.116.7:4010): protocol: write tcp 188.166.225.170:40036->45.91.116.7:4010: use of closed network connection; write tcp 188.166.225.170:40036->45.91.116.7:4010: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40036->45.91.116.7:4010: use of closed network connection Nov 5 18:52:35.899923829 (151h12m7.595772572s) [noise=true , address=77.90.31.18:4805 , piece_id=U2U34KK4AJK4A5HGNTHEYCIARLAR4X5NLOVCF63TN3PJUIRMMIHA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: write tcp 188.166.225.170:60758->77.90.31.18:4805: use of closed network connection; write tcp 188.166.225.170:60758->77.90.31.18:4805: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60758->77.90.31.18:4805: use of closed network connection Nov 5 18:52:35.900052084 (151h12m7.595900837s) [noise=true , address=77.90.31.16:4734 , piece_id=HFWVFM46PZFLZT6YNZVIA6HKLVN2PTV2GETZEKX356DMDGQO4HOA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: write tcp 188.166.225.170:50728->77.90.31.16:4734: use of closed network connection; write tcp 188.166.225.170:50728->77.90.31.16:4734: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50728->77.90.31.16:4734: use of closed network connection Nov 5 18:52:35.900231175 (151h12m7.596079938s) [noise=true , address=45.91.116.11:4117 , piece_id=QUGUYHEJWEEV733CBC3ONMV7XWDKVH2OFLQWXPQHPKZ7GTH4RLUA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: ecclient: upload failed (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, address:45.91.116.11:4117): protocol: write tcp 188.166.225.170:41292->45.91.116.11:4117: use of closed network connection; write tcp 188.166.225.170:41292->45.91.116.11:4117: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41292->45.91.116.11:4117: use of closed network connection Nov 5 18:52:35.900342116 (151h12m7.596190859s) [noise=true , address=77.90.31.15:4698 , piece_id=7VACL26N7ZCBXIA2I3XL5Q3UJ2BAHDIPXX4VZCQX3S63ABNW625Q, seg_pos={0 0}, upload=133242]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: write tcp 188.166.225.170:41118->77.90.31.15:4698: use of closed network connection; write tcp 188.166.225.170:41118->77.90.31.15:4698: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41118->77.90.31.15:4698: use of closed network connection Nov 5 18:52:35.900382814 (151h12m7.596231567s) [seg_pos={0 0}, upload=133242]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Nov 5 18:52:36.162442033 (151h12m7.858290786s) [upload=133242]: Done uploading segments... Nov 5 18:52:36.162481568 (151h12m7.858330321s) [upload=133242]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 5.897946188s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..............X...X.........................................] duration: 6.200442266s trace: https://jaeger.dev.storj.io/trace/74cf1e30f6d49950 FAIL (2025-11-05 19:22:34 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.200483674s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [..............X..X..........................................] duration: 5.743998686s trace: https://jaeger.dev.storj.io/trace/3e2ae8884260b157 FAIL (2025-11-05 19:22:34 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.744029676s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [...............XX...X.......................................] duration: 3.461703769s trace: https://jaeger.dev.storj.io/trace/204ab2570dc82d9c FAIL (2025-11-05 19:22:02 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.461739558s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.............X...X..........................................] duration: 5.933718397s trace: https://jaeger.dev.storj.io/trace/2e576aae48e8f03e FAIL (2025-11-05 19:21:55 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== 5 18:52:06.850167887 (151h11m38.54601663s) [noise=true , address=145.79.227.25:4618 , piece_id=XRUPY5BUPG7BXN2MI6MGYO4N3GW6TZDJE46MZFNRELWANHHG3N5A, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.850747301 (151h11m38.546596054s) [noise=true , address=77.90.31.15:4698 , piece_id=7VACL26N7ZCBXIA2I3XL5Q3UJ2BAHDIPXX4VZCQX3S63ABNW625Q, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.851222605 (151h11m38.547071358s) [noise=true , address=145.79.227.17:4972 , piece_id=H7UHQV7RDLCZUHY2XMXUAD2PX4ZBLWKTM47N6PH7UOGBRXPMWVOA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.851558575 (151h11m38.547407319s) [noise=true , address=145.79.227.22:4509 , piece_id=7RA5NJ72PI3DT4IC2ECVLZ3A2HXXL546ELVDXHZW4OXNN4D36OEA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.851909263 (151h11m38.547758006s) [noise=true , address=145.79.227.26:4654 , piece_id=E5J22KNPQSETSY443QQZXFFKUUJFHWFV22SQAVZ4BFQ4M6YWCNQQ, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.852327540 (151h11m38.548176293s) [noise=true , address=45.91.116.7:4010 , piece_id=EBVGGNQF3N4MTUIATJUWK4QZN7SEOWI6SIAW6N7JZOB5G6H33NEQ, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.852728624 (151h11m38.548577377s) [noise=true , address=145.79.227.9:4366 , piece_id=7ZRECKBQC4S6QLERBZY442WYS4H35XWOZYSGGPHSU7NQ6WL4UNVA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.853978774 (151h11m38.549827528s) [noise=true , address=77.90.31.18:4805 , piece_id=U2U34KK4AJK4A5HGNTHEYCIARLAR4X5NLOVCF63TN3PJUIRMMIHA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.854447296 (151h11m38.550296059s) [noise=true , address=77.90.31.16:4734 , piece_id=HFWVFM46PZFLZT6YNZVIA6HKLVN2PTV2GETZEKX356DMDGQO4HOA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.856523089 (151h11m38.552371842s) [noise=true , address=45.91.116.11:4117 , piece_id=QUGUYHEJWEEV733CBC3ONMV7XWDKVH2OFLQWXPQHPKZ7GTH4RLUA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.859490379 (151h11m38.555339122s) [noise=true , address=77.90.31.13:4628 , piece_id=QFTEHZ5FY7A5DPZZ44TRXGFRHPRNJZJHLQZDGRGOWYFEZYIZT2NQ, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.861350291 (151h11m38.557199034s) [noise=true , address=145.79.227.31:4797 , piece_id=2TFSZNURIAU573QUA5DUS7AJZNY2OXC6TJL3ST4VXIWY2WVB5YJA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.863368665 (151h11m38.559217418s) [noise=true , address=145.79.227.23:4547 , piece_id=K54H437KW5AAYUWOTMDBDP5GJ3ZABFKLAAOSS6WS5G5CROYEHODQ, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.864848794 (151h11m38.560697547s) [noise=true , address=145.79.227.24:4581 , piece_id=TSCVCH2KJEVMD4AVOEIHGTMRCRZ6NZMZA2CTQ6FUO2EIL3JAVYEA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.865644930 (151h11m38.561493683s) [noise=true , address=145.79.227.8:4330 , piece_id=4A47HIUX2MTJ2673XZZCMMF4T235TS4JWMNYJEVHHCKAHQX6APMA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:08.560504137 (151h11m40.25635289s) [upload=133242]: Next returned nil segment Nov 5 18:52:08.560522693 (151h11m40.256371446s) [upload=133242]: Waiting for error group managing segments... Nov 5 18:52:08.850333179 (151h11m40.546181932s) [noise=true , address=145.79.227.17:4972 , piece_id=H7UHQV7RDLCZUHY2XMXUAD2PX4ZBLWKTM47N6PH7UOGBRXPMWVOA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.853517822 (151h11m40.549366575s) [noise=true , address=145.79.227.22:4509 , piece_id=7RA5NJ72PI3DT4IC2ECVLZ3A2HXXL546ELVDXHZW4OXNN4D36OEA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.855547508 (151h11m40.551396261s) [noise=true , address=145.79.227.9:4366 , piece_id=7ZRECKBQC4S6QLERBZY442WYS4H35XWOZYSGGPHSU7NQ6WL4UNVA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.856103106 (151h11m40.551951859s) [noise=true , address=145.79.227.25:4618 , piece_id=XRUPY5BUPG7BXN2MI6MGYO4N3GW6TZDJE46MZFNRELWANHHG3N5A, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.860417321 (151h11m40.556266064s) [noise=true , address=145.79.227.8:4330 , piece_id=4A47HIUX2MTJ2673XZZCMMF4T235TS4JWMNYJEVHHCKAHQX6APMA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.865602796 (151h11m40.561451549s) [noise=true , address=145.79.227.23:4547 , piece_id=K54H437KW5AAYUWOTMDBDP5GJ3ZABFKLAAOSS6WS5G5CROYEHODQ, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.879393757 (151h11m40.57524251s) [noise=true , address=145.79.227.31:4797 , piece_id=2TFSZNURIAU573QUA5DUS7AJZNY2OXC6TJL3ST4VXIWY2WVB5YJA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.905541449 (151h11m40.601390202s) [noise=true , address=145.79.227.26:4654 , piece_id=E5J22KNPQSETSY443QQZXFFKUUJFHWFV22SQAVZ4BFQ4M6YWCNQQ, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.929018836 (151h11m40.624867589s) [noise=true , address=145.79.227.24:4581 , piece_id=TSCVCH2KJEVMD4AVOEIHGTMRCRZ6NZMZA2CTQ6FUO2EIL3JAVYEA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:35.899554917 (151h12m7.59540368s) [noise=true , address=77.90.31.17:4771 , piece_id=KVFE7VYW3RZJ25BWSH4BIRRQCMWDYOIK7EJ62XVREGN26YN7XYMQ, seg_pos={0 0}, upload=133242]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: write tcp 188.166.225.170:60676->77.90.31.17:4771: use of closed network connection; write tcp 188.166.225.170:60676->77.90.31.17:4771: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60676->77.90.31.17:4771: use of closed network connection Nov 5 18:52:35.899727506 (151h12m7.595576259s) [noise=true , address=77.90.31.13:4628 , piece_id=QFTEHZ5FY7A5DPZZ44TRXGFRHPRNJZJHLQZDGRGOWYFEZYIZT2NQ, seg_pos={0 0}, upload=133242]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: write tcp 188.166.225.170:42026->77.90.31.13:4628: use of closed network connection; write tcp 188.166.225.170:42026->77.90.31.13:4628: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42026->77.90.31.13:4628: use of closed network connection Nov 5 18:52:35.899860439 (151h12m7.595709192s) [noise=true , address=45.91.116.7:4010 , piece_id=EBVGGNQF3N4MTUIATJUWK4QZN7SEOWI6SIAW6N7JZOB5G6H33NEQ, seg_pos={0 0}, upload=133242]: Done uploading piece. err: ecclient: upload failed (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, address:45.91.116.7:4010): protocol: write tcp 188.166.225.170:40036->45.91.116.7:4010: use of closed network connection; write tcp 188.166.225.170:40036->45.91.116.7:4010: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40036->45.91.116.7:4010: use of closed network connection Nov 5 18:52:35.899923829 (151h12m7.595772572s) [noise=true , address=77.90.31.18:4805 , piece_id=U2U34KK4AJK4A5HGNTHEYCIARLAR4X5NLOVCF63TN3PJUIRMMIHA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: write tcp 188.166.225.170:60758->77.90.31.18:4805: use of closed network connection; write tcp 188.166.225.170:60758->77.90.31.18:4805: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60758->77.90.31.18:4805: use of closed network connection Nov 5 18:52:35.900052084 (151h12m7.595900837s) [noise=true , address=77.90.31.16:4734 , piece_id=HFWVFM46PZFLZT6YNZVIA6HKLVN2PTV2GETZEKX356DMDGQO4HOA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: write tcp 188.166.225.170:50728->77.90.31.16:4734: use of closed network connection; write tcp 188.166.225.170:50728->77.90.31.16:4734: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50728->77.90.31.16:4734: use of closed network connection Nov 5 18:52:35.900231175 (151h12m7.596079938s) [noise=true , address=45.91.116.11:4117 , piece_id=QUGUYHEJWEEV733CBC3ONMV7XWDKVH2OFLQWXPQHPKZ7GTH4RLUA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: ecclient: upload failed (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, address:45.91.116.11:4117): protocol: write tcp 188.166.225.170:41292->45.91.116.11:4117: use of closed network connection; write tcp 188.166.225.170:41292->45.91.116.11:4117: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41292->45.91.116.11:4117: use of closed network connection Nov 5 18:52:35.900342116 (151h12m7.596190859s) [noise=true , address=77.90.31.15:4698 , piece_id=7VACL26N7ZCBXIA2I3XL5Q3UJ2BAHDIPXX4VZCQX3S63ABNW625Q, seg_pos={0 0}, upload=133242]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: write tcp 188.166.225.170:41118->77.90.31.15:4698: use of closed network connection; write tcp 188.166.225.170:41118->77.90.31.15:4698: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41118->77.90.31.15:4698: use of closed network connection Nov 5 18:52:35.900382814 (151h12m7.596231567s) [seg_pos={0 0}, upload=133242]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Nov 5 18:52:36.162442033 (151h12m7.858290786s) [upload=133242]: Done uploading segments... Nov 5 18:52:36.162481568 (151h12m7.858330321s) [upload=133242]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 5.933754726s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [................XX...X......................................] duration: 3.439522394s trace: https://jaeger.dev.storj.io/trace/204ab2570dc82d9c FAIL (2025-11-05 19:21:41 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.439537572s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [...............X..X.........................................] duration: 5.701032341s trace: https://jaeger.dev.storj.io/trace/3e2ae8884260b157 FAIL (2025-11-05 19:21:40 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.70105858s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...............X...X........................................] duration: 6.382739743s trace: https://jaeger.dev.storj.io/trace/74cf1e30f6d49950 FAIL (2025-11-05 19:21:28 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.382768067s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [.................XX...X.....................................] duration: 3.441885704s trace: https://jaeger.dev.storj.io/trace/204ab2570dc82d9c FAIL (2025-11-05 19:21:13 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.441933615s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [................X...X.......................................] duration: 6.189992077s trace: https://jaeger.dev.storj.io/trace/74cf1e30f6d49950 FAIL (2025-11-05 19:21:09 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.190023547s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..............X...X.........................................] duration: 6.005558124s trace: https://jaeger.dev.storj.io/trace/2e576aae48e8f03e FAIL (2025-11-05 19:21:08 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== 5 18:52:06.850167887 (151h11m38.54601663s) [noise=true , address=145.79.227.25:4618 , piece_id=XRUPY5BUPG7BXN2MI6MGYO4N3GW6TZDJE46MZFNRELWANHHG3N5A, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.850747301 (151h11m38.546596054s) [noise=true , address=77.90.31.15:4698 , piece_id=7VACL26N7ZCBXIA2I3XL5Q3UJ2BAHDIPXX4VZCQX3S63ABNW625Q, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.851222605 (151h11m38.547071358s) [noise=true , address=145.79.227.17:4972 , piece_id=H7UHQV7RDLCZUHY2XMXUAD2PX4ZBLWKTM47N6PH7UOGBRXPMWVOA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.851558575 (151h11m38.547407319s) [noise=true , address=145.79.227.22:4509 , piece_id=7RA5NJ72PI3DT4IC2ECVLZ3A2HXXL546ELVDXHZW4OXNN4D36OEA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.851909263 (151h11m38.547758006s) [noise=true , address=145.79.227.26:4654 , piece_id=E5J22KNPQSETSY443QQZXFFKUUJFHWFV22SQAVZ4BFQ4M6YWCNQQ, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.852327540 (151h11m38.548176293s) [noise=true , address=45.91.116.7:4010 , piece_id=EBVGGNQF3N4MTUIATJUWK4QZN7SEOWI6SIAW6N7JZOB5G6H33NEQ, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.852728624 (151h11m38.548577377s) [noise=true , address=145.79.227.9:4366 , piece_id=7ZRECKBQC4S6QLERBZY442WYS4H35XWOZYSGGPHSU7NQ6WL4UNVA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.853978774 (151h11m38.549827528s) [noise=true , address=77.90.31.18:4805 , piece_id=U2U34KK4AJK4A5HGNTHEYCIARLAR4X5NLOVCF63TN3PJUIRMMIHA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.854447296 (151h11m38.550296059s) [noise=true , address=77.90.31.16:4734 , piece_id=HFWVFM46PZFLZT6YNZVIA6HKLVN2PTV2GETZEKX356DMDGQO4HOA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.856523089 (151h11m38.552371842s) [noise=true , address=45.91.116.11:4117 , piece_id=QUGUYHEJWEEV733CBC3ONMV7XWDKVH2OFLQWXPQHPKZ7GTH4RLUA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.859490379 (151h11m38.555339122s) [noise=true , address=77.90.31.13:4628 , piece_id=QFTEHZ5FY7A5DPZZ44TRXGFRHPRNJZJHLQZDGRGOWYFEZYIZT2NQ, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.861350291 (151h11m38.557199034s) [noise=true , address=145.79.227.31:4797 , piece_id=2TFSZNURIAU573QUA5DUS7AJZNY2OXC6TJL3ST4VXIWY2WVB5YJA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.863368665 (151h11m38.559217418s) [noise=true , address=145.79.227.23:4547 , piece_id=K54H437KW5AAYUWOTMDBDP5GJ3ZABFKLAAOSS6WS5G5CROYEHODQ, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.864848794 (151h11m38.560697547s) [noise=true , address=145.79.227.24:4581 , piece_id=TSCVCH2KJEVMD4AVOEIHGTMRCRZ6NZMZA2CTQ6FUO2EIL3JAVYEA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.865644930 (151h11m38.561493683s) [noise=true , address=145.79.227.8:4330 , piece_id=4A47HIUX2MTJ2673XZZCMMF4T235TS4JWMNYJEVHHCKAHQX6APMA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:08.560504137 (151h11m40.25635289s) [upload=133242]: Next returned nil segment Nov 5 18:52:08.560522693 (151h11m40.256371446s) [upload=133242]: Waiting for error group managing segments... Nov 5 18:52:08.850333179 (151h11m40.546181932s) [noise=true , address=145.79.227.17:4972 , piece_id=H7UHQV7RDLCZUHY2XMXUAD2PX4ZBLWKTM47N6PH7UOGBRXPMWVOA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.853517822 (151h11m40.549366575s) [noise=true , address=145.79.227.22:4509 , piece_id=7RA5NJ72PI3DT4IC2ECVLZ3A2HXXL546ELVDXHZW4OXNN4D36OEA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.855547508 (151h11m40.551396261s) [noise=true , address=145.79.227.9:4366 , piece_id=7ZRECKBQC4S6QLERBZY442WYS4H35XWOZYSGGPHSU7NQ6WL4UNVA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.856103106 (151h11m40.551951859s) [noise=true , address=145.79.227.25:4618 , piece_id=XRUPY5BUPG7BXN2MI6MGYO4N3GW6TZDJE46MZFNRELWANHHG3N5A, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.860417321 (151h11m40.556266064s) [noise=true , address=145.79.227.8:4330 , piece_id=4A47HIUX2MTJ2673XZZCMMF4T235TS4JWMNYJEVHHCKAHQX6APMA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.865602796 (151h11m40.561451549s) [noise=true , address=145.79.227.23:4547 , piece_id=K54H437KW5AAYUWOTMDBDP5GJ3ZABFKLAAOSS6WS5G5CROYEHODQ, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.879393757 (151h11m40.57524251s) [noise=true , address=145.79.227.31:4797 , piece_id=2TFSZNURIAU573QUA5DUS7AJZNY2OXC6TJL3ST4VXIWY2WVB5YJA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.905541449 (151h11m40.601390202s) [noise=true , address=145.79.227.26:4654 , piece_id=E5J22KNPQSETSY443QQZXFFKUUJFHWFV22SQAVZ4BFQ4M6YWCNQQ, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.929018836 (151h11m40.624867589s) [noise=true , address=145.79.227.24:4581 , piece_id=TSCVCH2KJEVMD4AVOEIHGTMRCRZ6NZMZA2CTQ6FUO2EIL3JAVYEA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:35.899554917 (151h12m7.59540368s) [noise=true , address=77.90.31.17:4771 , piece_id=KVFE7VYW3RZJ25BWSH4BIRRQCMWDYOIK7EJ62XVREGN26YN7XYMQ, seg_pos={0 0}, upload=133242]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: write tcp 188.166.225.170:60676->77.90.31.17:4771: use of closed network connection; write tcp 188.166.225.170:60676->77.90.31.17:4771: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60676->77.90.31.17:4771: use of closed network connection Nov 5 18:52:35.899727506 (151h12m7.595576259s) [noise=true , address=77.90.31.13:4628 , piece_id=QFTEHZ5FY7A5DPZZ44TRXGFRHPRNJZJHLQZDGRGOWYFEZYIZT2NQ, seg_pos={0 0}, upload=133242]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: write tcp 188.166.225.170:42026->77.90.31.13:4628: use of closed network connection; write tcp 188.166.225.170:42026->77.90.31.13:4628: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42026->77.90.31.13:4628: use of closed network connection Nov 5 18:52:35.899860439 (151h12m7.595709192s) [noise=true , address=45.91.116.7:4010 , piece_id=EBVGGNQF3N4MTUIATJUWK4QZN7SEOWI6SIAW6N7JZOB5G6H33NEQ, seg_pos={0 0}, upload=133242]: Done uploading piece. err: ecclient: upload failed (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, address:45.91.116.7:4010): protocol: write tcp 188.166.225.170:40036->45.91.116.7:4010: use of closed network connection; write tcp 188.166.225.170:40036->45.91.116.7:4010: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40036->45.91.116.7:4010: use of closed network connection Nov 5 18:52:35.899923829 (151h12m7.595772572s) [noise=true , address=77.90.31.18:4805 , piece_id=U2U34KK4AJK4A5HGNTHEYCIARLAR4X5NLOVCF63TN3PJUIRMMIHA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: write tcp 188.166.225.170:60758->77.90.31.18:4805: use of closed network connection; write tcp 188.166.225.170:60758->77.90.31.18:4805: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60758->77.90.31.18:4805: use of closed network connection Nov 5 18:52:35.900052084 (151h12m7.595900837s) [noise=true , address=77.90.31.16:4734 , piece_id=HFWVFM46PZFLZT6YNZVIA6HKLVN2PTV2GETZEKX356DMDGQO4HOA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: write tcp 188.166.225.170:50728->77.90.31.16:4734: use of closed network connection; write tcp 188.166.225.170:50728->77.90.31.16:4734: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50728->77.90.31.16:4734: use of closed network connection Nov 5 18:52:35.900231175 (151h12m7.596079938s) [noise=true , address=45.91.116.11:4117 , piece_id=QUGUYHEJWEEV733CBC3ONMV7XWDKVH2OFLQWXPQHPKZ7GTH4RLUA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: ecclient: upload failed (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, address:45.91.116.11:4117): protocol: write tcp 188.166.225.170:41292->45.91.116.11:4117: use of closed network connection; write tcp 188.166.225.170:41292->45.91.116.11:4117: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41292->45.91.116.11:4117: use of closed network connection Nov 5 18:52:35.900342116 (151h12m7.596190859s) [noise=true , address=77.90.31.15:4698 , piece_id=7VACL26N7ZCBXIA2I3XL5Q3UJ2BAHDIPXX4VZCQX3S63ABNW625Q, seg_pos={0 0}, upload=133242]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: write tcp 188.166.225.170:41118->77.90.31.15:4698: use of closed network connection; write tcp 188.166.225.170:41118->77.90.31.15:4698: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41118->77.90.31.15:4698: use of closed network connection Nov 5 18:52:35.900382814 (151h12m7.596231567s) [seg_pos={0 0}, upload=133242]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Nov 5 18:52:36.162442033 (151h12m7.858290786s) [upload=133242]: Done uploading segments... Nov 5 18:52:36.162481568 (151h12m7.858330321s) [upload=133242]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 6.005591597s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.................X...X......................................] duration: 6.167565194s trace: https://jaeger.dev.storj.io/trace/74cf1e30f6d49950 FAIL (2025-11-05 19:20:48 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.167595892s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [................X..X........................................] duration: 5.685469867s trace: https://jaeger.dev.storj.io/trace/3e2ae8884260b157 FAIL (2025-11-05 19:20:48 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.685564237s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...............X...X........................................] duration: 5.974943119s trace: https://jaeger.dev.storj.io/trace/2e576aae48e8f03e FAIL (2025-11-05 19:20:38 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== 5 18:52:06.850167887 (151h11m38.54601663s) [noise=true , address=145.79.227.25:4618 , piece_id=XRUPY5BUPG7BXN2MI6MGYO4N3GW6TZDJE46MZFNRELWANHHG3N5A, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.850747301 (151h11m38.546596054s) [noise=true , address=77.90.31.15:4698 , piece_id=7VACL26N7ZCBXIA2I3XL5Q3UJ2BAHDIPXX4VZCQX3S63ABNW625Q, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.851222605 (151h11m38.547071358s) [noise=true , address=145.79.227.17:4972 , piece_id=H7UHQV7RDLCZUHY2XMXUAD2PX4ZBLWKTM47N6PH7UOGBRXPMWVOA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.851558575 (151h11m38.547407319s) [noise=true , address=145.79.227.22:4509 , piece_id=7RA5NJ72PI3DT4IC2ECVLZ3A2HXXL546ELVDXHZW4OXNN4D36OEA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.851909263 (151h11m38.547758006s) [noise=true , address=145.79.227.26:4654 , piece_id=E5J22KNPQSETSY443QQZXFFKUUJFHWFV22SQAVZ4BFQ4M6YWCNQQ, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.852327540 (151h11m38.548176293s) [noise=true , address=45.91.116.7:4010 , piece_id=EBVGGNQF3N4MTUIATJUWK4QZN7SEOWI6SIAW6N7JZOB5G6H33NEQ, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.852728624 (151h11m38.548577377s) [noise=true , address=145.79.227.9:4366 , piece_id=7ZRECKBQC4S6QLERBZY442WYS4H35XWOZYSGGPHSU7NQ6WL4UNVA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.853978774 (151h11m38.549827528s) [noise=true , address=77.90.31.18:4805 , piece_id=U2U34KK4AJK4A5HGNTHEYCIARLAR4X5NLOVCF63TN3PJUIRMMIHA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.854447296 (151h11m38.550296059s) [noise=true , address=77.90.31.16:4734 , piece_id=HFWVFM46PZFLZT6YNZVIA6HKLVN2PTV2GETZEKX356DMDGQO4HOA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.856523089 (151h11m38.552371842s) [noise=true , address=45.91.116.11:4117 , piece_id=QUGUYHEJWEEV733CBC3ONMV7XWDKVH2OFLQWXPQHPKZ7GTH4RLUA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.859490379 (151h11m38.555339122s) [noise=true , address=77.90.31.13:4628 , piece_id=QFTEHZ5FY7A5DPZZ44TRXGFRHPRNJZJHLQZDGRGOWYFEZYIZT2NQ, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.861350291 (151h11m38.557199034s) [noise=true , address=145.79.227.31:4797 , piece_id=2TFSZNURIAU573QUA5DUS7AJZNY2OXC6TJL3ST4VXIWY2WVB5YJA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.863368665 (151h11m38.559217418s) [noise=true , address=145.79.227.23:4547 , piece_id=K54H437KW5AAYUWOTMDBDP5GJ3ZABFKLAAOSS6WS5G5CROYEHODQ, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.864848794 (151h11m38.560697547s) [noise=true , address=145.79.227.24:4581 , piece_id=TSCVCH2KJEVMD4AVOEIHGTMRCRZ6NZMZA2CTQ6FUO2EIL3JAVYEA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.865644930 (151h11m38.561493683s) [noise=true , address=145.79.227.8:4330 , piece_id=4A47HIUX2MTJ2673XZZCMMF4T235TS4JWMNYJEVHHCKAHQX6APMA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:08.560504137 (151h11m40.25635289s) [upload=133242]: Next returned nil segment Nov 5 18:52:08.560522693 (151h11m40.256371446s) [upload=133242]: Waiting for error group managing segments... Nov 5 18:52:08.850333179 (151h11m40.546181932s) [noise=true , address=145.79.227.17:4972 , piece_id=H7UHQV7RDLCZUHY2XMXUAD2PX4ZBLWKTM47N6PH7UOGBRXPMWVOA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.853517822 (151h11m40.549366575s) [noise=true , address=145.79.227.22:4509 , piece_id=7RA5NJ72PI3DT4IC2ECVLZ3A2HXXL546ELVDXHZW4OXNN4D36OEA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.855547508 (151h11m40.551396261s) [noise=true , address=145.79.227.9:4366 , piece_id=7ZRECKBQC4S6QLERBZY442WYS4H35XWOZYSGGPHSU7NQ6WL4UNVA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.856103106 (151h11m40.551951859s) [noise=true , address=145.79.227.25:4618 , piece_id=XRUPY5BUPG7BXN2MI6MGYO4N3GW6TZDJE46MZFNRELWANHHG3N5A, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.860417321 (151h11m40.556266064s) [noise=true , address=145.79.227.8:4330 , piece_id=4A47HIUX2MTJ2673XZZCMMF4T235TS4JWMNYJEVHHCKAHQX6APMA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.865602796 (151h11m40.561451549s) [noise=true , address=145.79.227.23:4547 , piece_id=K54H437KW5AAYUWOTMDBDP5GJ3ZABFKLAAOSS6WS5G5CROYEHODQ, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.879393757 (151h11m40.57524251s) [noise=true , address=145.79.227.31:4797 , piece_id=2TFSZNURIAU573QUA5DUS7AJZNY2OXC6TJL3ST4VXIWY2WVB5YJA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.905541449 (151h11m40.601390202s) [noise=true , address=145.79.227.26:4654 , piece_id=E5J22KNPQSETSY443QQZXFFKUUJFHWFV22SQAVZ4BFQ4M6YWCNQQ, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.929018836 (151h11m40.624867589s) [noise=true , address=145.79.227.24:4581 , piece_id=TSCVCH2KJEVMD4AVOEIHGTMRCRZ6NZMZA2CTQ6FUO2EIL3JAVYEA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:35.899554917 (151h12m7.59540368s) [noise=true , address=77.90.31.17:4771 , piece_id=KVFE7VYW3RZJ25BWSH4BIRRQCMWDYOIK7EJ62XVREGN26YN7XYMQ, seg_pos={0 0}, upload=133242]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: write tcp 188.166.225.170:60676->77.90.31.17:4771: use of closed network connection; write tcp 188.166.225.170:60676->77.90.31.17:4771: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60676->77.90.31.17:4771: use of closed network connection Nov 5 18:52:35.899727506 (151h12m7.595576259s) [noise=true , address=77.90.31.13:4628 , piece_id=QFTEHZ5FY7A5DPZZ44TRXGFRHPRNJZJHLQZDGRGOWYFEZYIZT2NQ, seg_pos={0 0}, upload=133242]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: write tcp 188.166.225.170:42026->77.90.31.13:4628: use of closed network connection; write tcp 188.166.225.170:42026->77.90.31.13:4628: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42026->77.90.31.13:4628: use of closed network connection Nov 5 18:52:35.899860439 (151h12m7.595709192s) [noise=true , address=45.91.116.7:4010 , piece_id=EBVGGNQF3N4MTUIATJUWK4QZN7SEOWI6SIAW6N7JZOB5G6H33NEQ, seg_pos={0 0}, upload=133242]: Done uploading piece. err: ecclient: upload failed (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, address:45.91.116.7:4010): protocol: write tcp 188.166.225.170:40036->45.91.116.7:4010: use of closed network connection; write tcp 188.166.225.170:40036->45.91.116.7:4010: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40036->45.91.116.7:4010: use of closed network connection Nov 5 18:52:35.899923829 (151h12m7.595772572s) [noise=true , address=77.90.31.18:4805 , piece_id=U2U34KK4AJK4A5HGNTHEYCIARLAR4X5NLOVCF63TN3PJUIRMMIHA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: write tcp 188.166.225.170:60758->77.90.31.18:4805: use of closed network connection; write tcp 188.166.225.170:60758->77.90.31.18:4805: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60758->77.90.31.18:4805: use of closed network connection Nov 5 18:52:35.900052084 (151h12m7.595900837s) [noise=true , address=77.90.31.16:4734 , piece_id=HFWVFM46PZFLZT6YNZVIA6HKLVN2PTV2GETZEKX356DMDGQO4HOA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: write tcp 188.166.225.170:50728->77.90.31.16:4734: use of closed network connection; write tcp 188.166.225.170:50728->77.90.31.16:4734: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50728->77.90.31.16:4734: use of closed network connection Nov 5 18:52:35.900231175 (151h12m7.596079938s) [noise=true , address=45.91.116.11:4117 , piece_id=QUGUYHEJWEEV733CBC3ONMV7XWDKVH2OFLQWXPQHPKZ7GTH4RLUA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: ecclient: upload failed (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, address:45.91.116.11:4117): protocol: write tcp 188.166.225.170:41292->45.91.116.11:4117: use of closed network connection; write tcp 188.166.225.170:41292->45.91.116.11:4117: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41292->45.91.116.11:4117: use of closed network connection Nov 5 18:52:35.900342116 (151h12m7.596190859s) [noise=true , address=77.90.31.15:4698 , piece_id=7VACL26N7ZCBXIA2I3XL5Q3UJ2BAHDIPXX4VZCQX3S63ABNW625Q, seg_pos={0 0}, upload=133242]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: write tcp 188.166.225.170:41118->77.90.31.15:4698: use of closed network connection; write tcp 188.166.225.170:41118->77.90.31.15:4698: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41118->77.90.31.15:4698: use of closed network connection Nov 5 18:52:35.900382814 (151h12m7.596231567s) [seg_pos={0 0}, upload=133242]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Nov 5 18:52:36.162442033 (151h12m7.858290786s) [upload=133242]: Done uploading segments... Nov 5 18:52:36.162481568 (151h12m7.858330321s) [upload=133242]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 5.974972095s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [..................XX...X....................................] duration: 3.455332304s trace: https://jaeger.dev.storj.io/trace/204ab2570dc82d9c FAIL (2025-11-05 19:20:16 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.455423196s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [................X...X.......................................] duration: 6.223903781s trace: https://jaeger.dev.storj.io/trace/2e576aae48e8f03e FAIL (2025-11-05 19:20:08 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== 5 18:52:06.850167887 (151h11m38.54601663s) [noise=true , address=145.79.227.25:4618 , piece_id=XRUPY5BUPG7BXN2MI6MGYO4N3GW6TZDJE46MZFNRELWANHHG3N5A, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.850747301 (151h11m38.546596054s) [noise=true , address=77.90.31.15:4698 , piece_id=7VACL26N7ZCBXIA2I3XL5Q3UJ2BAHDIPXX4VZCQX3S63ABNW625Q, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.851222605 (151h11m38.547071358s) [noise=true , address=145.79.227.17:4972 , piece_id=H7UHQV7RDLCZUHY2XMXUAD2PX4ZBLWKTM47N6PH7UOGBRXPMWVOA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.851558575 (151h11m38.547407319s) [noise=true , address=145.79.227.22:4509 , piece_id=7RA5NJ72PI3DT4IC2ECVLZ3A2HXXL546ELVDXHZW4OXNN4D36OEA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.851909263 (151h11m38.547758006s) [noise=true , address=145.79.227.26:4654 , piece_id=E5J22KNPQSETSY443QQZXFFKUUJFHWFV22SQAVZ4BFQ4M6YWCNQQ, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.852327540 (151h11m38.548176293s) [noise=true , address=45.91.116.7:4010 , piece_id=EBVGGNQF3N4MTUIATJUWK4QZN7SEOWI6SIAW6N7JZOB5G6H33NEQ, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.852728624 (151h11m38.548577377s) [noise=true , address=145.79.227.9:4366 , piece_id=7ZRECKBQC4S6QLERBZY442WYS4H35XWOZYSGGPHSU7NQ6WL4UNVA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.853978774 (151h11m38.549827528s) [noise=true , address=77.90.31.18:4805 , piece_id=U2U34KK4AJK4A5HGNTHEYCIARLAR4X5NLOVCF63TN3PJUIRMMIHA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.854447296 (151h11m38.550296059s) [noise=true , address=77.90.31.16:4734 , piece_id=HFWVFM46PZFLZT6YNZVIA6HKLVN2PTV2GETZEKX356DMDGQO4HOA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.856523089 (151h11m38.552371842s) [noise=true , address=45.91.116.11:4117 , piece_id=QUGUYHEJWEEV733CBC3ONMV7XWDKVH2OFLQWXPQHPKZ7GTH4RLUA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.859490379 (151h11m38.555339122s) [noise=true , address=77.90.31.13:4628 , piece_id=QFTEHZ5FY7A5DPZZ44TRXGFRHPRNJZJHLQZDGRGOWYFEZYIZT2NQ, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.861350291 (151h11m38.557199034s) [noise=true , address=145.79.227.31:4797 , piece_id=2TFSZNURIAU573QUA5DUS7AJZNY2OXC6TJL3ST4VXIWY2WVB5YJA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.863368665 (151h11m38.559217418s) [noise=true , address=145.79.227.23:4547 , piece_id=K54H437KW5AAYUWOTMDBDP5GJ3ZABFKLAAOSS6WS5G5CROYEHODQ, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.864848794 (151h11m38.560697547s) [noise=true , address=145.79.227.24:4581 , piece_id=TSCVCH2KJEVMD4AVOEIHGTMRCRZ6NZMZA2CTQ6FUO2EIL3JAVYEA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.865644930 (151h11m38.561493683s) [noise=true , address=145.79.227.8:4330 , piece_id=4A47HIUX2MTJ2673XZZCMMF4T235TS4JWMNYJEVHHCKAHQX6APMA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:08.560504137 (151h11m40.25635289s) [upload=133242]: Next returned nil segment Nov 5 18:52:08.560522693 (151h11m40.256371446s) [upload=133242]: Waiting for error group managing segments... Nov 5 18:52:08.850333179 (151h11m40.546181932s) [noise=true , address=145.79.227.17:4972 , piece_id=H7UHQV7RDLCZUHY2XMXUAD2PX4ZBLWKTM47N6PH7UOGBRXPMWVOA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.853517822 (151h11m40.549366575s) [noise=true , address=145.79.227.22:4509 , piece_id=7RA5NJ72PI3DT4IC2ECVLZ3A2HXXL546ELVDXHZW4OXNN4D36OEA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.855547508 (151h11m40.551396261s) [noise=true , address=145.79.227.9:4366 , piece_id=7ZRECKBQC4S6QLERBZY442WYS4H35XWOZYSGGPHSU7NQ6WL4UNVA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.856103106 (151h11m40.551951859s) [noise=true , address=145.79.227.25:4618 , piece_id=XRUPY5BUPG7BXN2MI6MGYO4N3GW6TZDJE46MZFNRELWANHHG3N5A, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.860417321 (151h11m40.556266064s) [noise=true , address=145.79.227.8:4330 , piece_id=4A47HIUX2MTJ2673XZZCMMF4T235TS4JWMNYJEVHHCKAHQX6APMA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.865602796 (151h11m40.561451549s) [noise=true , address=145.79.227.23:4547 , piece_id=K54H437KW5AAYUWOTMDBDP5GJ3ZABFKLAAOSS6WS5G5CROYEHODQ, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.879393757 (151h11m40.57524251s) [noise=true , address=145.79.227.31:4797 , piece_id=2TFSZNURIAU573QUA5DUS7AJZNY2OXC6TJL3ST4VXIWY2WVB5YJA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.905541449 (151h11m40.601390202s) [noise=true , address=145.79.227.26:4654 , piece_id=E5J22KNPQSETSY443QQZXFFKUUJFHWFV22SQAVZ4BFQ4M6YWCNQQ, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.929018836 (151h11m40.624867589s) [noise=true , address=145.79.227.24:4581 , piece_id=TSCVCH2KJEVMD4AVOEIHGTMRCRZ6NZMZA2CTQ6FUO2EIL3JAVYEA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:35.899554917 (151h12m7.59540368s) [noise=true , address=77.90.31.17:4771 , piece_id=KVFE7VYW3RZJ25BWSH4BIRRQCMWDYOIK7EJ62XVREGN26YN7XYMQ, seg_pos={0 0}, upload=133242]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: write tcp 188.166.225.170:60676->77.90.31.17:4771: use of closed network connection; write tcp 188.166.225.170:60676->77.90.31.17:4771: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60676->77.90.31.17:4771: use of closed network connection Nov 5 18:52:35.899727506 (151h12m7.595576259s) [noise=true , address=77.90.31.13:4628 , piece_id=QFTEHZ5FY7A5DPZZ44TRXGFRHPRNJZJHLQZDGRGOWYFEZYIZT2NQ, seg_pos={0 0}, upload=133242]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: write tcp 188.166.225.170:42026->77.90.31.13:4628: use of closed network connection; write tcp 188.166.225.170:42026->77.90.31.13:4628: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42026->77.90.31.13:4628: use of closed network connection Nov 5 18:52:35.899860439 (151h12m7.595709192s) [noise=true , address=45.91.116.7:4010 , piece_id=EBVGGNQF3N4MTUIATJUWK4QZN7SEOWI6SIAW6N7JZOB5G6H33NEQ, seg_pos={0 0}, upload=133242]: Done uploading piece. err: ecclient: upload failed (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, address:45.91.116.7:4010): protocol: write tcp 188.166.225.170:40036->45.91.116.7:4010: use of closed network connection; write tcp 188.166.225.170:40036->45.91.116.7:4010: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40036->45.91.116.7:4010: use of closed network connection Nov 5 18:52:35.899923829 (151h12m7.595772572s) [noise=true , address=77.90.31.18:4805 , piece_id=U2U34KK4AJK4A5HGNTHEYCIARLAR4X5NLOVCF63TN3PJUIRMMIHA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: write tcp 188.166.225.170:60758->77.90.31.18:4805: use of closed network connection; write tcp 188.166.225.170:60758->77.90.31.18:4805: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60758->77.90.31.18:4805: use of closed network connection Nov 5 18:52:35.900052084 (151h12m7.595900837s) [noise=true , address=77.90.31.16:4734 , piece_id=HFWVFM46PZFLZT6YNZVIA6HKLVN2PTV2GETZEKX356DMDGQO4HOA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: write tcp 188.166.225.170:50728->77.90.31.16:4734: use of closed network connection; write tcp 188.166.225.170:50728->77.90.31.16:4734: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50728->77.90.31.16:4734: use of closed network connection Nov 5 18:52:35.900231175 (151h12m7.596079938s) [noise=true , address=45.91.116.11:4117 , piece_id=QUGUYHEJWEEV733CBC3ONMV7XWDKVH2OFLQWXPQHPKZ7GTH4RLUA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: ecclient: upload failed (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, address:45.91.116.11:4117): protocol: write tcp 188.166.225.170:41292->45.91.116.11:4117: use of closed network connection; write tcp 188.166.225.170:41292->45.91.116.11:4117: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41292->45.91.116.11:4117: use of closed network connection Nov 5 18:52:35.900342116 (151h12m7.596190859s) [noise=true , address=77.90.31.15:4698 , piece_id=7VACL26N7ZCBXIA2I3XL5Q3UJ2BAHDIPXX4VZCQX3S63ABNW625Q, seg_pos={0 0}, upload=133242]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: write tcp 188.166.225.170:41118->77.90.31.15:4698: use of closed network connection; write tcp 188.166.225.170:41118->77.90.31.15:4698: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41118->77.90.31.15:4698: use of closed network connection Nov 5 18:52:35.900382814 (151h12m7.596231567s) [seg_pos={0 0}, upload=133242]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Nov 5 18:52:36.162442033 (151h12m7.858290786s) [upload=133242]: Done uploading segments... Nov 5 18:52:36.162481568 (151h12m7.858330321s) [upload=133242]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 6.224146392s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..................X...X.....................................] duration: 6.482270482s trace: https://jaeger.dev.storj.io/trace/74cf1e30f6d49950 FAIL (2025-11-05 19:20:03 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.482298696s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [...................XX...X...................................] duration: 3.484475389s trace: https://jaeger.dev.storj.io/trace/204ab2570dc82d9c FAIL (2025-11-05 19:20:01 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.484508751s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)