Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [....................................................X...X...] duration: 5.852275457s trace: https://jaeger.dev.storj.io/trace/2e576aae48e8f03e FAIL (2025-11-05 18:53:51 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== 5 18:52:06.850167887 (151h11m38.54601663s) [noise=true , address=145.79.227.25:4618 , piece_id=XRUPY5BUPG7BXN2MI6MGYO4N3GW6TZDJE46MZFNRELWANHHG3N5A, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.850747301 (151h11m38.546596054s) [noise=true , address=77.90.31.15:4698 , piece_id=7VACL26N7ZCBXIA2I3XL5Q3UJ2BAHDIPXX4VZCQX3S63ABNW625Q, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.851222605 (151h11m38.547071358s) [noise=true , address=145.79.227.17:4972 , piece_id=H7UHQV7RDLCZUHY2XMXUAD2PX4ZBLWKTM47N6PH7UOGBRXPMWVOA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.851558575 (151h11m38.547407319s) [noise=true , address=145.79.227.22:4509 , piece_id=7RA5NJ72PI3DT4IC2ECVLZ3A2HXXL546ELVDXHZW4OXNN4D36OEA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.851909263 (151h11m38.547758006s) [noise=true , address=145.79.227.26:4654 , piece_id=E5J22KNPQSETSY443QQZXFFKUUJFHWFV22SQAVZ4BFQ4M6YWCNQQ, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.852327540 (151h11m38.548176293s) [noise=true , address=45.91.116.7:4010 , piece_id=EBVGGNQF3N4MTUIATJUWK4QZN7SEOWI6SIAW6N7JZOB5G6H33NEQ, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.852728624 (151h11m38.548577377s) [noise=true , address=145.79.227.9:4366 , piece_id=7ZRECKBQC4S6QLERBZY442WYS4H35XWOZYSGGPHSU7NQ6WL4UNVA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.853978774 (151h11m38.549827528s) [noise=true , address=77.90.31.18:4805 , piece_id=U2U34KK4AJK4A5HGNTHEYCIARLAR4X5NLOVCF63TN3PJUIRMMIHA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.854447296 (151h11m38.550296059s) [noise=true , address=77.90.31.16:4734 , piece_id=HFWVFM46PZFLZT6YNZVIA6HKLVN2PTV2GETZEKX356DMDGQO4HOA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.856523089 (151h11m38.552371842s) [noise=true , address=45.91.116.11:4117 , piece_id=QUGUYHEJWEEV733CBC3ONMV7XWDKVH2OFLQWXPQHPKZ7GTH4RLUA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.859490379 (151h11m38.555339122s) [noise=true , address=77.90.31.13:4628 , piece_id=QFTEHZ5FY7A5DPZZ44TRXGFRHPRNJZJHLQZDGRGOWYFEZYIZT2NQ, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.861350291 (151h11m38.557199034s) [noise=true , address=145.79.227.31:4797 , piece_id=2TFSZNURIAU573QUA5DUS7AJZNY2OXC6TJL3ST4VXIWY2WVB5YJA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.863368665 (151h11m38.559217418s) [noise=true , address=145.79.227.23:4547 , piece_id=K54H437KW5AAYUWOTMDBDP5GJ3ZABFKLAAOSS6WS5G5CROYEHODQ, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.864848794 (151h11m38.560697547s) [noise=true , address=145.79.227.24:4581 , piece_id=TSCVCH2KJEVMD4AVOEIHGTMRCRZ6NZMZA2CTQ6FUO2EIL3JAVYEA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.865644930 (151h11m38.561493683s) [noise=true , address=145.79.227.8:4330 , piece_id=4A47HIUX2MTJ2673XZZCMMF4T235TS4JWMNYJEVHHCKAHQX6APMA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:08.560504137 (151h11m40.25635289s) [upload=133242]: Next returned nil segment Nov 5 18:52:08.560522693 (151h11m40.256371446s) [upload=133242]: Waiting for error group managing segments... Nov 5 18:52:08.850333179 (151h11m40.546181932s) [noise=true , address=145.79.227.17:4972 , piece_id=H7UHQV7RDLCZUHY2XMXUAD2PX4ZBLWKTM47N6PH7UOGBRXPMWVOA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.853517822 (151h11m40.549366575s) [noise=true , address=145.79.227.22:4509 , piece_id=7RA5NJ72PI3DT4IC2ECVLZ3A2HXXL546ELVDXHZW4OXNN4D36OEA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.855547508 (151h11m40.551396261s) [noise=true , address=145.79.227.9:4366 , piece_id=7ZRECKBQC4S6QLERBZY442WYS4H35XWOZYSGGPHSU7NQ6WL4UNVA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.856103106 (151h11m40.551951859s) [noise=true , address=145.79.227.25:4618 , piece_id=XRUPY5BUPG7BXN2MI6MGYO4N3GW6TZDJE46MZFNRELWANHHG3N5A, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.860417321 (151h11m40.556266064s) [noise=true , address=145.79.227.8:4330 , piece_id=4A47HIUX2MTJ2673XZZCMMF4T235TS4JWMNYJEVHHCKAHQX6APMA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.865602796 (151h11m40.561451549s) [noise=true , address=145.79.227.23:4547 , piece_id=K54H437KW5AAYUWOTMDBDP5GJ3ZABFKLAAOSS6WS5G5CROYEHODQ, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.879393757 (151h11m40.57524251s) [noise=true , address=145.79.227.31:4797 , piece_id=2TFSZNURIAU573QUA5DUS7AJZNY2OXC6TJL3ST4VXIWY2WVB5YJA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.905541449 (151h11m40.601390202s) [noise=true , address=145.79.227.26:4654 , piece_id=E5J22KNPQSETSY443QQZXFFKUUJFHWFV22SQAVZ4BFQ4M6YWCNQQ, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.929018836 (151h11m40.624867589s) [noise=true , address=145.79.227.24:4581 , piece_id=TSCVCH2KJEVMD4AVOEIHGTMRCRZ6NZMZA2CTQ6FUO2EIL3JAVYEA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:35.899554917 (151h12m7.59540368s) [noise=true , address=77.90.31.17:4771 , piece_id=KVFE7VYW3RZJ25BWSH4BIRRQCMWDYOIK7EJ62XVREGN26YN7XYMQ, seg_pos={0 0}, upload=133242]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: write tcp 188.166.225.170:60676->77.90.31.17:4771: use of closed network connection; write tcp 188.166.225.170:60676->77.90.31.17:4771: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60676->77.90.31.17:4771: use of closed network connection Nov 5 18:52:35.899727506 (151h12m7.595576259s) [noise=true , address=77.90.31.13:4628 , piece_id=QFTEHZ5FY7A5DPZZ44TRXGFRHPRNJZJHLQZDGRGOWYFEZYIZT2NQ, seg_pos={0 0}, upload=133242]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: write tcp 188.166.225.170:42026->77.90.31.13:4628: use of closed network connection; write tcp 188.166.225.170:42026->77.90.31.13:4628: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42026->77.90.31.13:4628: use of closed network connection Nov 5 18:52:35.899860439 (151h12m7.595709192s) [noise=true , address=45.91.116.7:4010 , piece_id=EBVGGNQF3N4MTUIATJUWK4QZN7SEOWI6SIAW6N7JZOB5G6H33NEQ, seg_pos={0 0}, upload=133242]: Done uploading piece. err: ecclient: upload failed (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, address:45.91.116.7:4010): protocol: write tcp 188.166.225.170:40036->45.91.116.7:4010: use of closed network connection; write tcp 188.166.225.170:40036->45.91.116.7:4010: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40036->45.91.116.7:4010: use of closed network connection Nov 5 18:52:35.899923829 (151h12m7.595772572s) [noise=true , address=77.90.31.18:4805 , piece_id=U2U34KK4AJK4A5HGNTHEYCIARLAR4X5NLOVCF63TN3PJUIRMMIHA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: write tcp 188.166.225.170:60758->77.90.31.18:4805: use of closed network connection; write tcp 188.166.225.170:60758->77.90.31.18:4805: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60758->77.90.31.18:4805: use of closed network connection Nov 5 18:52:35.900052084 (151h12m7.595900837s) [noise=true , address=77.90.31.16:4734 , piece_id=HFWVFM46PZFLZT6YNZVIA6HKLVN2PTV2GETZEKX356DMDGQO4HOA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: write tcp 188.166.225.170:50728->77.90.31.16:4734: use of closed network connection; write tcp 188.166.225.170:50728->77.90.31.16:4734: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50728->77.90.31.16:4734: use of closed network connection Nov 5 18:52:35.900231175 (151h12m7.596079938s) [noise=true , address=45.91.116.11:4117 , piece_id=QUGUYHEJWEEV733CBC3ONMV7XWDKVH2OFLQWXPQHPKZ7GTH4RLUA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: ecclient: upload failed (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, address:45.91.116.11:4117): protocol: write tcp 188.166.225.170:41292->45.91.116.11:4117: use of closed network connection; write tcp 188.166.225.170:41292->45.91.116.11:4117: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41292->45.91.116.11:4117: use of closed network connection Nov 5 18:52:35.900342116 (151h12m7.596190859s) [noise=true , address=77.90.31.15:4698 , piece_id=7VACL26N7ZCBXIA2I3XL5Q3UJ2BAHDIPXX4VZCQX3S63ABNW625Q, seg_pos={0 0}, upload=133242]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: write tcp 188.166.225.170:41118->77.90.31.15:4698: use of closed network connection; write tcp 188.166.225.170:41118->77.90.31.15:4698: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41118->77.90.31.15:4698: use of closed network connection Nov 5 18:52:35.900382814 (151h12m7.596231567s) [seg_pos={0 0}, upload=133242]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Nov 5 18:52:36.162442033 (151h12m7.858290786s) [upload=133242]: Done uploading segments... Nov 5 18:52:36.162481568 (151h12m7.858330321s) [upload=133242]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 5.852306386s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....................................................X...X..] duration: 6.366363717s trace: https://jaeger.dev.storj.io/trace/74cf1e30f6d49950 FAIL (2025-11-05 18:53:41 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.366390957s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [......................................................X..X..] duration: 5.772440115s trace: https://jaeger.dev.storj.io/trace/3e2ae8884260b157 FAIL (2025-11-05 18:53:15 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.772463569s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....................................................X...X..] duration: 6.035223815s trace: https://jaeger.dev.storj.io/trace/2e576aae48e8f03e FAIL (2025-11-05 18:53:02 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== 5 18:52:06.850167887 (151h11m38.54601663s) [noise=true , address=145.79.227.25:4618 , piece_id=XRUPY5BUPG7BXN2MI6MGYO4N3GW6TZDJE46MZFNRELWANHHG3N5A, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.850747301 (151h11m38.546596054s) [noise=true , address=77.90.31.15:4698 , piece_id=7VACL26N7ZCBXIA2I3XL5Q3UJ2BAHDIPXX4VZCQX3S63ABNW625Q, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.851222605 (151h11m38.547071358s) [noise=true , address=145.79.227.17:4972 , piece_id=H7UHQV7RDLCZUHY2XMXUAD2PX4ZBLWKTM47N6PH7UOGBRXPMWVOA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.851558575 (151h11m38.547407319s) [noise=true , address=145.79.227.22:4509 , piece_id=7RA5NJ72PI3DT4IC2ECVLZ3A2HXXL546ELVDXHZW4OXNN4D36OEA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.851909263 (151h11m38.547758006s) [noise=true , address=145.79.227.26:4654 , piece_id=E5J22KNPQSETSY443QQZXFFKUUJFHWFV22SQAVZ4BFQ4M6YWCNQQ, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.852327540 (151h11m38.548176293s) [noise=true , address=45.91.116.7:4010 , piece_id=EBVGGNQF3N4MTUIATJUWK4QZN7SEOWI6SIAW6N7JZOB5G6H33NEQ, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.852728624 (151h11m38.548577377s) [noise=true , address=145.79.227.9:4366 , piece_id=7ZRECKBQC4S6QLERBZY442WYS4H35XWOZYSGGPHSU7NQ6WL4UNVA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.853978774 (151h11m38.549827528s) [noise=true , address=77.90.31.18:4805 , piece_id=U2U34KK4AJK4A5HGNTHEYCIARLAR4X5NLOVCF63TN3PJUIRMMIHA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.854447296 (151h11m38.550296059s) [noise=true , address=77.90.31.16:4734 , piece_id=HFWVFM46PZFLZT6YNZVIA6HKLVN2PTV2GETZEKX356DMDGQO4HOA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.856523089 (151h11m38.552371842s) [noise=true , address=45.91.116.11:4117 , piece_id=QUGUYHEJWEEV733CBC3ONMV7XWDKVH2OFLQWXPQHPKZ7GTH4RLUA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.859490379 (151h11m38.555339122s) [noise=true , address=77.90.31.13:4628 , piece_id=QFTEHZ5FY7A5DPZZ44TRXGFRHPRNJZJHLQZDGRGOWYFEZYIZT2NQ, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.861350291 (151h11m38.557199034s) [noise=true , address=145.79.227.31:4797 , piece_id=2TFSZNURIAU573QUA5DUS7AJZNY2OXC6TJL3ST4VXIWY2WVB5YJA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.863368665 (151h11m38.559217418s) [noise=true , address=145.79.227.23:4547 , piece_id=K54H437KW5AAYUWOTMDBDP5GJ3ZABFKLAAOSS6WS5G5CROYEHODQ, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.864848794 (151h11m38.560697547s) [noise=true , address=145.79.227.24:4581 , piece_id=TSCVCH2KJEVMD4AVOEIHGTMRCRZ6NZMZA2CTQ6FUO2EIL3JAVYEA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.865644930 (151h11m38.561493683s) [noise=true , address=145.79.227.8:4330 , piece_id=4A47HIUX2MTJ2673XZZCMMF4T235TS4JWMNYJEVHHCKAHQX6APMA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:08.560504137 (151h11m40.25635289s) [upload=133242]: Next returned nil segment Nov 5 18:52:08.560522693 (151h11m40.256371446s) [upload=133242]: Waiting for error group managing segments... Nov 5 18:52:08.850333179 (151h11m40.546181932s) [noise=true , address=145.79.227.17:4972 , piece_id=H7UHQV7RDLCZUHY2XMXUAD2PX4ZBLWKTM47N6PH7UOGBRXPMWVOA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.853517822 (151h11m40.549366575s) [noise=true , address=145.79.227.22:4509 , piece_id=7RA5NJ72PI3DT4IC2ECVLZ3A2HXXL546ELVDXHZW4OXNN4D36OEA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.855547508 (151h11m40.551396261s) [noise=true , address=145.79.227.9:4366 , piece_id=7ZRECKBQC4S6QLERBZY442WYS4H35XWOZYSGGPHSU7NQ6WL4UNVA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.856103106 (151h11m40.551951859s) [noise=true , address=145.79.227.25:4618 , piece_id=XRUPY5BUPG7BXN2MI6MGYO4N3GW6TZDJE46MZFNRELWANHHG3N5A, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.860417321 (151h11m40.556266064s) [noise=true , address=145.79.227.8:4330 , piece_id=4A47HIUX2MTJ2673XZZCMMF4T235TS4JWMNYJEVHHCKAHQX6APMA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.865602796 (151h11m40.561451549s) [noise=true , address=145.79.227.23:4547 , piece_id=K54H437KW5AAYUWOTMDBDP5GJ3ZABFKLAAOSS6WS5G5CROYEHODQ, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.879393757 (151h11m40.57524251s) [noise=true , address=145.79.227.31:4797 , piece_id=2TFSZNURIAU573QUA5DUS7AJZNY2OXC6TJL3ST4VXIWY2WVB5YJA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.905541449 (151h11m40.601390202s) [noise=true , address=145.79.227.26:4654 , piece_id=E5J22KNPQSETSY443QQZXFFKUUJFHWFV22SQAVZ4BFQ4M6YWCNQQ, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.929018836 (151h11m40.624867589s) [noise=true , address=145.79.227.24:4581 , piece_id=TSCVCH2KJEVMD4AVOEIHGTMRCRZ6NZMZA2CTQ6FUO2EIL3JAVYEA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:35.899554917 (151h12m7.59540368s) [noise=true , address=77.90.31.17:4771 , piece_id=KVFE7VYW3RZJ25BWSH4BIRRQCMWDYOIK7EJ62XVREGN26YN7XYMQ, seg_pos={0 0}, upload=133242]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: write tcp 188.166.225.170:60676->77.90.31.17:4771: use of closed network connection; write tcp 188.166.225.170:60676->77.90.31.17:4771: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60676->77.90.31.17:4771: use of closed network connection Nov 5 18:52:35.899727506 (151h12m7.595576259s) [noise=true , address=77.90.31.13:4628 , piece_id=QFTEHZ5FY7A5DPZZ44TRXGFRHPRNJZJHLQZDGRGOWYFEZYIZT2NQ, seg_pos={0 0}, upload=133242]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: write tcp 188.166.225.170:42026->77.90.31.13:4628: use of closed network connection; write tcp 188.166.225.170:42026->77.90.31.13:4628: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42026->77.90.31.13:4628: use of closed network connection Nov 5 18:52:35.899860439 (151h12m7.595709192s) [noise=true , address=45.91.116.7:4010 , piece_id=EBVGGNQF3N4MTUIATJUWK4QZN7SEOWI6SIAW6N7JZOB5G6H33NEQ, seg_pos={0 0}, upload=133242]: Done uploading piece. err: ecclient: upload failed (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, address:45.91.116.7:4010): protocol: write tcp 188.166.225.170:40036->45.91.116.7:4010: use of closed network connection; write tcp 188.166.225.170:40036->45.91.116.7:4010: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40036->45.91.116.7:4010: use of closed network connection Nov 5 18:52:35.899923829 (151h12m7.595772572s) [noise=true , address=77.90.31.18:4805 , piece_id=U2U34KK4AJK4A5HGNTHEYCIARLAR4X5NLOVCF63TN3PJUIRMMIHA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: write tcp 188.166.225.170:60758->77.90.31.18:4805: use of closed network connection; write tcp 188.166.225.170:60758->77.90.31.18:4805: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60758->77.90.31.18:4805: use of closed network connection Nov 5 18:52:35.900052084 (151h12m7.595900837s) [noise=true , address=77.90.31.16:4734 , piece_id=HFWVFM46PZFLZT6YNZVIA6HKLVN2PTV2GETZEKX356DMDGQO4HOA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: write tcp 188.166.225.170:50728->77.90.31.16:4734: use of closed network connection; write tcp 188.166.225.170:50728->77.90.31.16:4734: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50728->77.90.31.16:4734: use of closed network connection Nov 5 18:52:35.900231175 (151h12m7.596079938s) [noise=true , address=45.91.116.11:4117 , piece_id=QUGUYHEJWEEV733CBC3ONMV7XWDKVH2OFLQWXPQHPKZ7GTH4RLUA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: ecclient: upload failed (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, address:45.91.116.11:4117): protocol: write tcp 188.166.225.170:41292->45.91.116.11:4117: use of closed network connection; write tcp 188.166.225.170:41292->45.91.116.11:4117: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41292->45.91.116.11:4117: use of closed network connection Nov 5 18:52:35.900342116 (151h12m7.596190859s) [noise=true , address=77.90.31.15:4698 , piece_id=7VACL26N7ZCBXIA2I3XL5Q3UJ2BAHDIPXX4VZCQX3S63ABNW625Q, seg_pos={0 0}, upload=133242]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: write tcp 188.166.225.170:41118->77.90.31.15:4698: use of closed network connection; write tcp 188.166.225.170:41118->77.90.31.15:4698: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41118->77.90.31.15:4698: use of closed network connection Nov 5 18:52:35.900382814 (151h12m7.596231567s) [seg_pos={0 0}, upload=133242]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Nov 5 18:52:36.162442033 (151h12m7.858290786s) [upload=133242]: Done uploading segments... Nov 5 18:52:36.162481568 (151h12m7.858330321s) [upload=133242]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 6.035280623s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [.....................................................XX...X.] duration: 3.456493221s trace: https://jaeger.dev.storj.io/trace/204ab2570dc82d9c FAIL (2025-11-05 18:53:02 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.456515132s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [......................................................X...X.] duration: 6.524054761s trace: https://jaeger.dev.storj.io/trace/74cf1e30f6d49950 FAIL (2025-11-05 18:52:44 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.524084337s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [......................................................X...X.] duration: 6.632603408s trace: https://jaeger.dev.storj.io/trace/2e576aae48e8f03e FAIL (2025-11-05 18:52:43 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== 5 18:52:06.850167887 (151h11m38.54601663s) [noise=true , address=145.79.227.25:4618 , piece_id=XRUPY5BUPG7BXN2MI6MGYO4N3GW6TZDJE46MZFNRELWANHHG3N5A, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.850747301 (151h11m38.546596054s) [noise=true , address=77.90.31.15:4698 , piece_id=7VACL26N7ZCBXIA2I3XL5Q3UJ2BAHDIPXX4VZCQX3S63ABNW625Q, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.851222605 (151h11m38.547071358s) [noise=true , address=145.79.227.17:4972 , piece_id=H7UHQV7RDLCZUHY2XMXUAD2PX4ZBLWKTM47N6PH7UOGBRXPMWVOA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.851558575 (151h11m38.547407319s) [noise=true , address=145.79.227.22:4509 , piece_id=7RA5NJ72PI3DT4IC2ECVLZ3A2HXXL546ELVDXHZW4OXNN4D36OEA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.851909263 (151h11m38.547758006s) [noise=true , address=145.79.227.26:4654 , piece_id=E5J22KNPQSETSY443QQZXFFKUUJFHWFV22SQAVZ4BFQ4M6YWCNQQ, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.852327540 (151h11m38.548176293s) [noise=true , address=45.91.116.7:4010 , piece_id=EBVGGNQF3N4MTUIATJUWK4QZN7SEOWI6SIAW6N7JZOB5G6H33NEQ, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.852728624 (151h11m38.548577377s) [noise=true , address=145.79.227.9:4366 , piece_id=7ZRECKBQC4S6QLERBZY442WYS4H35XWOZYSGGPHSU7NQ6WL4UNVA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.853978774 (151h11m38.549827528s) [noise=true , address=77.90.31.18:4805 , piece_id=U2U34KK4AJK4A5HGNTHEYCIARLAR4X5NLOVCF63TN3PJUIRMMIHA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.854447296 (151h11m38.550296059s) [noise=true , address=77.90.31.16:4734 , piece_id=HFWVFM46PZFLZT6YNZVIA6HKLVN2PTV2GETZEKX356DMDGQO4HOA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.856523089 (151h11m38.552371842s) [noise=true , address=45.91.116.11:4117 , piece_id=QUGUYHEJWEEV733CBC3ONMV7XWDKVH2OFLQWXPQHPKZ7GTH4RLUA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.859490379 (151h11m38.555339122s) [noise=true , address=77.90.31.13:4628 , piece_id=QFTEHZ5FY7A5DPZZ44TRXGFRHPRNJZJHLQZDGRGOWYFEZYIZT2NQ, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.861350291 (151h11m38.557199034s) [noise=true , address=145.79.227.31:4797 , piece_id=2TFSZNURIAU573QUA5DUS7AJZNY2OXC6TJL3ST4VXIWY2WVB5YJA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.863368665 (151h11m38.559217418s) [noise=true , address=145.79.227.23:4547 , piece_id=K54H437KW5AAYUWOTMDBDP5GJ3ZABFKLAAOSS6WS5G5CROYEHODQ, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.864848794 (151h11m38.560697547s) [noise=true , address=145.79.227.24:4581 , piece_id=TSCVCH2KJEVMD4AVOEIHGTMRCRZ6NZMZA2CTQ6FUO2EIL3JAVYEA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.865644930 (151h11m38.561493683s) [noise=true , address=145.79.227.8:4330 , piece_id=4A47HIUX2MTJ2673XZZCMMF4T235TS4JWMNYJEVHHCKAHQX6APMA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:08.560504137 (151h11m40.25635289s) [upload=133242]: Next returned nil segment Nov 5 18:52:08.560522693 (151h11m40.256371446s) [upload=133242]: Waiting for error group managing segments... Nov 5 18:52:08.850333179 (151h11m40.546181932s) [noise=true , address=145.79.227.17:4972 , piece_id=H7UHQV7RDLCZUHY2XMXUAD2PX4ZBLWKTM47N6PH7UOGBRXPMWVOA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.853517822 (151h11m40.549366575s) [noise=true , address=145.79.227.22:4509 , piece_id=7RA5NJ72PI3DT4IC2ECVLZ3A2HXXL546ELVDXHZW4OXNN4D36OEA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.855547508 (151h11m40.551396261s) [noise=true , address=145.79.227.9:4366 , piece_id=7ZRECKBQC4S6QLERBZY442WYS4H35XWOZYSGGPHSU7NQ6WL4UNVA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.856103106 (151h11m40.551951859s) [noise=true , address=145.79.227.25:4618 , piece_id=XRUPY5BUPG7BXN2MI6MGYO4N3GW6TZDJE46MZFNRELWANHHG3N5A, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.860417321 (151h11m40.556266064s) [noise=true , address=145.79.227.8:4330 , piece_id=4A47HIUX2MTJ2673XZZCMMF4T235TS4JWMNYJEVHHCKAHQX6APMA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.865602796 (151h11m40.561451549s) [noise=true , address=145.79.227.23:4547 , piece_id=K54H437KW5AAYUWOTMDBDP5GJ3ZABFKLAAOSS6WS5G5CROYEHODQ, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.879393757 (151h11m40.57524251s) [noise=true , address=145.79.227.31:4797 , piece_id=2TFSZNURIAU573QUA5DUS7AJZNY2OXC6TJL3ST4VXIWY2WVB5YJA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.905541449 (151h11m40.601390202s) [noise=true , address=145.79.227.26:4654 , piece_id=E5J22KNPQSETSY443QQZXFFKUUJFHWFV22SQAVZ4BFQ4M6YWCNQQ, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.929018836 (151h11m40.624867589s) [noise=true , address=145.79.227.24:4581 , piece_id=TSCVCH2KJEVMD4AVOEIHGTMRCRZ6NZMZA2CTQ6FUO2EIL3JAVYEA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:35.899554917 (151h12m7.59540368s) [noise=true , address=77.90.31.17:4771 , piece_id=KVFE7VYW3RZJ25BWSH4BIRRQCMWDYOIK7EJ62XVREGN26YN7XYMQ, seg_pos={0 0}, upload=133242]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: write tcp 188.166.225.170:60676->77.90.31.17:4771: use of closed network connection; write tcp 188.166.225.170:60676->77.90.31.17:4771: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60676->77.90.31.17:4771: use of closed network connection Nov 5 18:52:35.899727506 (151h12m7.595576259s) [noise=true , address=77.90.31.13:4628 , piece_id=QFTEHZ5FY7A5DPZZ44TRXGFRHPRNJZJHLQZDGRGOWYFEZYIZT2NQ, seg_pos={0 0}, upload=133242]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: write tcp 188.166.225.170:42026->77.90.31.13:4628: use of closed network connection; write tcp 188.166.225.170:42026->77.90.31.13:4628: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42026->77.90.31.13:4628: use of closed network connection Nov 5 18:52:35.899860439 (151h12m7.595709192s) [noise=true , address=45.91.116.7:4010 , piece_id=EBVGGNQF3N4MTUIATJUWK4QZN7SEOWI6SIAW6N7JZOB5G6H33NEQ, seg_pos={0 0}, upload=133242]: Done uploading piece. err: ecclient: upload failed (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, address:45.91.116.7:4010): protocol: write tcp 188.166.225.170:40036->45.91.116.7:4010: use of closed network connection; write tcp 188.166.225.170:40036->45.91.116.7:4010: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40036->45.91.116.7:4010: use of closed network connection Nov 5 18:52:35.899923829 (151h12m7.595772572s) [noise=true , address=77.90.31.18:4805 , piece_id=U2U34KK4AJK4A5HGNTHEYCIARLAR4X5NLOVCF63TN3PJUIRMMIHA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: write tcp 188.166.225.170:60758->77.90.31.18:4805: use of closed network connection; write tcp 188.166.225.170:60758->77.90.31.18:4805: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60758->77.90.31.18:4805: use of closed network connection Nov 5 18:52:35.900052084 (151h12m7.595900837s) [noise=true , address=77.90.31.16:4734 , piece_id=HFWVFM46PZFLZT6YNZVIA6HKLVN2PTV2GETZEKX356DMDGQO4HOA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: write tcp 188.166.225.170:50728->77.90.31.16:4734: use of closed network connection; write tcp 188.166.225.170:50728->77.90.31.16:4734: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50728->77.90.31.16:4734: use of closed network connection Nov 5 18:52:35.900231175 (151h12m7.596079938s) [noise=true , address=45.91.116.11:4117 , piece_id=QUGUYHEJWEEV733CBC3ONMV7XWDKVH2OFLQWXPQHPKZ7GTH4RLUA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: ecclient: upload failed (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, address:45.91.116.11:4117): protocol: write tcp 188.166.225.170:41292->45.91.116.11:4117: use of closed network connection; write tcp 188.166.225.170:41292->45.91.116.11:4117: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41292->45.91.116.11:4117: use of closed network connection Nov 5 18:52:35.900342116 (151h12m7.596190859s) [noise=true , address=77.90.31.15:4698 , piece_id=7VACL26N7ZCBXIA2I3XL5Q3UJ2BAHDIPXX4VZCQX3S63ABNW625Q, seg_pos={0 0}, upload=133242]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: write tcp 188.166.225.170:41118->77.90.31.15:4698: use of closed network connection; write tcp 188.166.225.170:41118->77.90.31.15:4698: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41118->77.90.31.15:4698: use of closed network connection Nov 5 18:52:35.900382814 (151h12m7.596231567s) [seg_pos={0 0}, upload=133242]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Nov 5 18:52:36.162442033 (151h12m7.858290786s) [upload=133242]: Done uploading segments... Nov 5 18:52:36.162481568 (151h12m7.858330321s) [upload=133242]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 6.632618537s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......................................................X...X] duration: 30.265580379s trace: https://jaeger.dev.storj.io/trace/2e576aae48e8f03e FAIL (2025-11-05 18:52:36 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== 5 18:52:06.850167887 (151h11m38.54601663s) [noise=true , address=145.79.227.25:4618 , piece_id=XRUPY5BUPG7BXN2MI6MGYO4N3GW6TZDJE46MZFNRELWANHHG3N5A, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.850747301 (151h11m38.546596054s) [noise=true , address=77.90.31.15:4698 , piece_id=7VACL26N7ZCBXIA2I3XL5Q3UJ2BAHDIPXX4VZCQX3S63ABNW625Q, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.851222605 (151h11m38.547071358s) [noise=true , address=145.79.227.17:4972 , piece_id=H7UHQV7RDLCZUHY2XMXUAD2PX4ZBLWKTM47N6PH7UOGBRXPMWVOA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.851558575 (151h11m38.547407319s) [noise=true , address=145.79.227.22:4509 , piece_id=7RA5NJ72PI3DT4IC2ECVLZ3A2HXXL546ELVDXHZW4OXNN4D36OEA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.851909263 (151h11m38.547758006s) [noise=true , address=145.79.227.26:4654 , piece_id=E5J22KNPQSETSY443QQZXFFKUUJFHWFV22SQAVZ4BFQ4M6YWCNQQ, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.852327540 (151h11m38.548176293s) [noise=true , address=45.91.116.7:4010 , piece_id=EBVGGNQF3N4MTUIATJUWK4QZN7SEOWI6SIAW6N7JZOB5G6H33NEQ, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.852728624 (151h11m38.548577377s) [noise=true , address=145.79.227.9:4366 , piece_id=7ZRECKBQC4S6QLERBZY442WYS4H35XWOZYSGGPHSU7NQ6WL4UNVA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.853978774 (151h11m38.549827528s) [noise=true , address=77.90.31.18:4805 , piece_id=U2U34KK4AJK4A5HGNTHEYCIARLAR4X5NLOVCF63TN3PJUIRMMIHA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.854447296 (151h11m38.550296059s) [noise=true , address=77.90.31.16:4734 , piece_id=HFWVFM46PZFLZT6YNZVIA6HKLVN2PTV2GETZEKX356DMDGQO4HOA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.856523089 (151h11m38.552371842s) [noise=true , address=45.91.116.11:4117 , piece_id=QUGUYHEJWEEV733CBC3ONMV7XWDKVH2OFLQWXPQHPKZ7GTH4RLUA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.859490379 (151h11m38.555339122s) [noise=true , address=77.90.31.13:4628 , piece_id=QFTEHZ5FY7A5DPZZ44TRXGFRHPRNJZJHLQZDGRGOWYFEZYIZT2NQ, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.861350291 (151h11m38.557199034s) [noise=true , address=145.79.227.31:4797 , piece_id=2TFSZNURIAU573QUA5DUS7AJZNY2OXC6TJL3ST4VXIWY2WVB5YJA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.863368665 (151h11m38.559217418s) [noise=true , address=145.79.227.23:4547 , piece_id=K54H437KW5AAYUWOTMDBDP5GJ3ZABFKLAAOSS6WS5G5CROYEHODQ, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.864848794 (151h11m38.560697547s) [noise=true , address=145.79.227.24:4581 , piece_id=TSCVCH2KJEVMD4AVOEIHGTMRCRZ6NZMZA2CTQ6FUO2EIL3JAVYEA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.865644930 (151h11m38.561493683s) [noise=true , address=145.79.227.8:4330 , piece_id=4A47HIUX2MTJ2673XZZCMMF4T235TS4JWMNYJEVHHCKAHQX6APMA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:08.560504137 (151h11m40.25635289s) [upload=133242]: Next returned nil segment Nov 5 18:52:08.560522693 (151h11m40.256371446s) [upload=133242]: Waiting for error group managing segments... Nov 5 18:52:08.850333179 (151h11m40.546181932s) [noise=true , address=145.79.227.17:4972 , piece_id=H7UHQV7RDLCZUHY2XMXUAD2PX4ZBLWKTM47N6PH7UOGBRXPMWVOA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.853517822 (151h11m40.549366575s) [noise=true , address=145.79.227.22:4509 , piece_id=7RA5NJ72PI3DT4IC2ECVLZ3A2HXXL546ELVDXHZW4OXNN4D36OEA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.855547508 (151h11m40.551396261s) [noise=true , address=145.79.227.9:4366 , piece_id=7ZRECKBQC4S6QLERBZY442WYS4H35XWOZYSGGPHSU7NQ6WL4UNVA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.856103106 (151h11m40.551951859s) [noise=true , address=145.79.227.25:4618 , piece_id=XRUPY5BUPG7BXN2MI6MGYO4N3GW6TZDJE46MZFNRELWANHHG3N5A, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.860417321 (151h11m40.556266064s) [noise=true , address=145.79.227.8:4330 , piece_id=4A47HIUX2MTJ2673XZZCMMF4T235TS4JWMNYJEVHHCKAHQX6APMA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.865602796 (151h11m40.561451549s) [noise=true , address=145.79.227.23:4547 , piece_id=K54H437KW5AAYUWOTMDBDP5GJ3ZABFKLAAOSS6WS5G5CROYEHODQ, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.879393757 (151h11m40.57524251s) [noise=true , address=145.79.227.31:4797 , piece_id=2TFSZNURIAU573QUA5DUS7AJZNY2OXC6TJL3ST4VXIWY2WVB5YJA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.905541449 (151h11m40.601390202s) [noise=true , address=145.79.227.26:4654 , piece_id=E5J22KNPQSETSY443QQZXFFKUUJFHWFV22SQAVZ4BFQ4M6YWCNQQ, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.929018836 (151h11m40.624867589s) [noise=true , address=145.79.227.24:4581 , piece_id=TSCVCH2KJEVMD4AVOEIHGTMRCRZ6NZMZA2CTQ6FUO2EIL3JAVYEA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:35.899554917 (151h12m7.59540368s) [noise=true , address=77.90.31.17:4771 , piece_id=KVFE7VYW3RZJ25BWSH4BIRRQCMWDYOIK7EJ62XVREGN26YN7XYMQ, seg_pos={0 0}, upload=133242]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: write tcp 188.166.225.170:60676->77.90.31.17:4771: use of closed network connection; write tcp 188.166.225.170:60676->77.90.31.17:4771: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60676->77.90.31.17:4771: use of closed network connection Nov 5 18:52:35.899727506 (151h12m7.595576259s) [noise=true , address=77.90.31.13:4628 , piece_id=QFTEHZ5FY7A5DPZZ44TRXGFRHPRNJZJHLQZDGRGOWYFEZYIZT2NQ, seg_pos={0 0}, upload=133242]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: write tcp 188.166.225.170:42026->77.90.31.13:4628: use of closed network connection; write tcp 188.166.225.170:42026->77.90.31.13:4628: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42026->77.90.31.13:4628: use of closed network connection Nov 5 18:52:35.899860439 (151h12m7.595709192s) [noise=true , address=45.91.116.7:4010 , piece_id=EBVGGNQF3N4MTUIATJUWK4QZN7SEOWI6SIAW6N7JZOB5G6H33NEQ, seg_pos={0 0}, upload=133242]: Done uploading piece. err: ecclient: upload failed (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, address:45.91.116.7:4010): protocol: write tcp 188.166.225.170:40036->45.91.116.7:4010: use of closed network connection; write tcp 188.166.225.170:40036->45.91.116.7:4010: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40036->45.91.116.7:4010: use of closed network connection Nov 5 18:52:35.899923829 (151h12m7.595772572s) [noise=true , address=77.90.31.18:4805 , piece_id=U2U34KK4AJK4A5HGNTHEYCIARLAR4X5NLOVCF63TN3PJUIRMMIHA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: write tcp 188.166.225.170:60758->77.90.31.18:4805: use of closed network connection; write tcp 188.166.225.170:60758->77.90.31.18:4805: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60758->77.90.31.18:4805: use of closed network connection Nov 5 18:52:35.900052084 (151h12m7.595900837s) [noise=true , address=77.90.31.16:4734 , piece_id=HFWVFM46PZFLZT6YNZVIA6HKLVN2PTV2GETZEKX356DMDGQO4HOA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: write tcp 188.166.225.170:50728->77.90.31.16:4734: use of closed network connection; write tcp 188.166.225.170:50728->77.90.31.16:4734: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50728->77.90.31.16:4734: use of closed network connection Nov 5 18:52:35.900231175 (151h12m7.596079938s) [noise=true , address=45.91.116.11:4117 , piece_id=QUGUYHEJWEEV733CBC3ONMV7XWDKVH2OFLQWXPQHPKZ7GTH4RLUA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: ecclient: upload failed (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, address:45.91.116.11:4117): protocol: write tcp 188.166.225.170:41292->45.91.116.11:4117: use of closed network connection; write tcp 188.166.225.170:41292->45.91.116.11:4117: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41292->45.91.116.11:4117: use of closed network connection Nov 5 18:52:35.900342116 (151h12m7.596190859s) [noise=true , address=77.90.31.15:4698 , piece_id=7VACL26N7ZCBXIA2I3XL5Q3UJ2BAHDIPXX4VZCQX3S63ABNW625Q, seg_pos={0 0}, upload=133242]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: write tcp 188.166.225.170:41118->77.90.31.15:4698: use of closed network connection; write tcp 188.166.225.170:41118->77.90.31.15:4698: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41118->77.90.31.15:4698: use of closed network connection Nov 5 18:52:35.900382814 (151h12m7.596231567s) [seg_pos={0 0}, upload=133242]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Nov 5 18:52:36.162442033 (151h12m7.858290786s) [upload=133242]: Done uploading segments... Nov 5 18:52:36.162481568 (151h12m7.858330321s) [upload=133242]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 30.265617801s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.......................................................X..X.] duration: 8.033183918s trace: https://jaeger.dev.storj.io/trace/3e2ae8884260b157 FAIL (2025-11-05 18:52:30 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 8.033199828s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [........................................................X..X] duration: 1m0.001035713s trace: https://jaeger.dev.storj.io/trace/3e2ae8884260b157 FAIL (2025-11-05 18:52:22 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.001063845s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......................................................X...X] duration: 30.000874937s trace: https://jaeger.dev.storj.io/trace/74cf1e30f6d49950 FAIL (2025-11-05 18:52:21 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.000900615s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [......................................................XX...X] duration: 5.001945279s trace: https://jaeger.dev.storj.io/trace/204ab2570dc82d9c FAIL (2025-11-05 18:51:58 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001971929s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 6.223649215s trace: https://jaeger.dev.storj.io/trace/58e8c1d3b7ee8f0a FAIL (2025-11-05 18:51:26 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.223679674s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [.......................................................XX...] duration: 4.025240404s trace: https://jaeger.dev.storj.io/trace/3ea7c69f954229ec FAIL (2025-11-05 18:51:26 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.025290701s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 5.912766008s trace: https://jaeger.dev.storj.io/trace/4578568744f6a3d6 FAIL (2025-11-05 18:51:14 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 5 18:48:36.399583130 (151h8m8.095431873s) [upload=133198]: Starting upload Nov 5 18:48:36.399692318 (151h8m8.095541071s) [upload=133198]: Done starting upload Nov 5 18:48:36.399752994 (151h8m8.095601737s) [upload=133198]: Uploading segments... Nov 5 18:48:36.399936253 (151h8m8.095785007s) [upload=133198]: Got next segment. Inline: false Nov 5 18:48:36.399987891 (151h8m8.095836644s) [upload=133198]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 5 18:48:36.399993442 (151h8m8.095842195s) [upload=133198]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 5 18:48:36.647833063 (151h8m8.343681807s) [seg_pos={0 0}, upload=133198]: Begin upload segment... Nov 5 18:48:36.647875144 (151h8m8.343723887s) [seg_pos={0 0}, upload=133198]: Done begin upload segment. Nov 5 18:48:36.647901734 (151h8m8.343750487s) [noise=true , address=45.91.116.6:4974 , piece_id=VIZKH56GJABCBBBTA3W5CGSUS645JINBDPVR5YBTWOGQ77I5YZ4Q, seg_pos={0 0}, upload=133198]: Uploading piece... Nov 5 18:48:36.649986123 (151h8m8.345834867s) [noise=true , address=145.79.227.23:4545 , piece_id=2VRGKAVVXU3LRNTZZUCYLUFYSX2YCA7MAER33GCABLC4EFZAKP3A, seg_pos={0 0}, upload=133198]: Uploading piece... Nov 5 18:48:36.650653094 (151h8m8.346501847s) [noise=true , address=145.79.227.17:4972 , piece_id=AGKRIHAOAC37VCKRUO2MKC4BUP34BC3DL6OJ56JBVPNDAQL4C2UQ, seg_pos={0 0}, upload=133198]: Uploading piece... Nov 5 18:48:36.651122347 (151h8m8.3469711s) [noise=true , address=145.79.227.25:4618 , piece_id=C5GZJZMGBHWQTIPOYBLZTQPNHTKIEIN2TFQYZOH73QI3V6UKQMFQ, seg_pos={0 0}, upload=133198]: Uploading piece... Nov 5 18:48:36.651439301 (151h8m8.347288044s) [noise=true , address=77.90.31.14:4663 , piece_id=DZJYBZ2I24SBIKJNKVI5TPGDWMHGYKMNPCDOIDDM2MSPWIH4OYVQ, seg_pos={0 0}, upload=133198]: Uploading piece... Nov 5 18:48:36.652593810 (151h8m8.348442563s) [noise=true , address=145.79.227.16:4911 , piece_id=CPHIIOPUWLUE6B2VHM3XVOJZAVXICEZF4DNMDCDB5FZEIOJH2T4Q, seg_pos={0 0}, upload=133198]: Uploading piece... Nov 5 18:48:36.653013098 (151h8m8.348861851s) [noise=true , address=77.90.31.19:4842 , piece_id=XA47EB4PMODRDDVWIP2YJ2XYAGRHCMEDQTYFHQGUCJCOPPI6OT2Q, seg_pos={0 0}, upload=133198]: Uploading piece... Nov 5 18:48:36.653698693 (151h8m8.349547446s) [noise=true , address=145.79.227.11:4729 , piece_id=GSVCIJUKZUDXLMJP6O7E347MSST3TW53LHJVH5ACVPS46EPIUVWA, seg_pos={0 0}, upload=133198]: Uploading piece... Nov 5 18:48:36.654497464 (151h8m8.350346217s) [noise=true , address=145.79.227.14:4838 , piece_id=NU3PLF3XK4MJQVYPK5ORBSR5Z7KBIKQB4GSPR77TTRGH6HWVNPIQ, seg_pos={0 0}, upload=133198]: Uploading piece... Nov 5 18:48:36.654918918 (151h8m8.350767661s) [noise=true , address=145.79.227.31:4799 , piece_id=WA2DSLH4AH6AHWZUOANQ4U2PWGSSU4F6DQNY5YIG3FYKWNQOWYCQ, seg_pos={0 0}, upload=133198]: Uploading piece... Nov 5 18:48:36.656003563 (151h8m8.351852316s) [noise=true , address=145.79.227.27:4690 , piece_id=YFCXBO7UPR3JND6PFUSUPAV32WU3YUXJPPN62UE4KIBZHCWGXMMQ, seg_pos={0 0}, upload=133198]: Uploading piece... Nov 5 18:48:36.656332689 (151h8m8.352181442s) [noise=true , address=145.79.227.9:4367 , piece_id=UND6AUULHZL67V2ZUVMTUHRXV3I2RFGYOEOINE6ERKUAQSC6AXYQ, seg_pos={0 0}, upload=133198]: Uploading piece... Nov 5 18:48:36.657419229 (151h8m8.353267972s) [noise=true , address=77.90.31.12:4590 , piece_id=KF5M57SIYYH5WBQZZ3KXQINWBYIIFA7ZVP6QWJORHMKZFFDCJ3MQ, seg_pos={0 0}, upload=133198]: Uploading piece... Nov 5 18:48:36.658763309 (151h8m8.354612052s) [noise=true , address=145.79.227.20:4437 , piece_id=PTYH2CNZUY6VSRDA5HDVQ5YCAJD26J5GIUM6KXRRK3DPWTOBFBKA, seg_pos={0 0}, upload=133198]: Uploading piece... Nov 5 18:48:36.661246016 (151h8m8.357094769s) [noise=true , address=145.79.227.8:4331 , piece_id=OZGLAEOKHMCUZX2CEOB5AJCN7K6AUDUGYRWJUTQZEEW26OZVUDOA, seg_pos={0 0}, upload=133198]: Uploading piece... Nov 5 18:48:36.662727748 (151h8m8.358576501s) [noise=true , address=145.79.227.32:4834 , piece_id=MQAE3ZGOCAS2QSKHBR4SJO2UVDVKRQPHILI5JR7NE46JG6UYPI5Q, seg_pos={0 0}, upload=133198]: Uploading piece... Nov 5 18:48:38.266459954 (151h8m9.962308697s) [upload=133198]: Next returned nil segment Nov 5 18:48:38.266474912 (151h8m9.962323665s) [upload=133198]: Waiting for error group managing segments... Nov 5 18:48:38.648038210 (151h8m10.343886963s) [noise=true , address=145.79.227.17:4972 , piece_id=AGKRIHAOAC37VCKRUO2MKC4BUP34BC3DL6OJ56JBVPNDAQL4C2UQ, seg_pos={0 0}, upload=133198]: Done uploading piece. err: <nil> Nov 5 18:48:38.649794854 (151h8m10.345643607s) [noise=true , address=145.79.227.11:4729 , piece_id=GSVCIJUKZUDXLMJP6O7E347MSST3TW53LHJVH5ACVPS46EPIUVWA, seg_pos={0 0}, upload=133198]: Done uploading piece. err: <nil> Nov 5 18:48:38.653624727 (151h8m10.34947348s) [noise=true , address=145.79.227.32:4834 , piece_id=MQAE3ZGOCAS2QSKHBR4SJO2UVDVKRQPHILI5JR7NE46JG6UYPI5Q, seg_pos={0 0}, upload=133198]: Done uploading piece. err: <nil> Nov 5 18:48:38.654714322 (151h8m10.350563075s) [noise=true , address=145.79.227.8:4331 , piece_id=OZGLAEOKHMCUZX2CEOB5AJCN7K6AUDUGYRWJUTQZEEW26OZVUDOA, seg_pos={0 0}, upload=133198]: Done uploading piece. err: <nil> Nov 5 18:48:38.655492113 (151h8m10.351340856s) [noise=true , address=145.79.227.14:4838 , piece_id=NU3PLF3XK4MJQVYPK5ORBSR5Z7KBIKQB4GSPR77TTRGH6HWVNPIQ, seg_pos={0 0}, upload=133198]: Done uploading piece. err: <nil> Nov 5 18:48:38.656244135 (151h8m10.352092888s) [noise=true , address=145.79.227.16:4911 , piece_id=CPHIIOPUWLUE6B2VHM3XVOJZAVXICEZF4DNMDCDB5FZEIOJH2T4Q, seg_pos={0 0}, upload=133198]: Done uploading piece. err: <nil> Nov 5 18:48:38.656809301 (151h8m10.352658054s) [noise=true , address=145.79.227.25:4618 , piece_id=C5GZJZMGBHWQTIPOYBLZTQPNHTKIEIN2TFQYZOH73QI3V6UKQMFQ, seg_pos={0 0}, upload=133198]: Done uploading piece. err: <nil> Nov 5 18:48:38.659785509 (151h8m10.355634262s) [noise=true , address=145.79.227.20:4437 , piece_id=PTYH2CNZUY6VSRDA5HDVQ5YCAJD26J5GIUM6KXRRK3DPWTOBFBKA, seg_pos={0 0}, upload=133198]: Done uploading piece. err: <nil> Nov 5 18:48:38.662194085 (151h8m10.358042848s) [noise=true , address=145.79.227.27:4690 , piece_id=YFCXBO7UPR3JND6PFUSUPAV32WU3YUXJPPN62UE4KIBZHCWGXMMQ, seg_pos={0 0}, upload=133198]: Done uploading piece. err: <nil> Nov 5 18:48:38.708392088 (151h8m10.404240841s) [noise=true , address=145.79.227.9:4367 , piece_id=UND6AUULHZL67V2ZUVMTUHRXV3I2RFGYOEOINE6ERKUAQSC6AXYQ, seg_pos={0 0}, upload=133198]: Done uploading piece. err: <nil> Nov 5 18:48:38.728515969 (151h8m10.424364712s) [noise=true , address=145.79.227.31:4799 , piece_id=WA2DSLH4AH6AHWZUOANQ4U2PWGSSU4F6DQNY5YIG3FYKWNQOWYCQ, seg_pos={0 0}, upload=133198]: Done uploading piece. err: <nil> Nov 5 18:48:38.738513017 (151h8m10.43436176s) [noise=true , address=145.79.227.23:4545 , piece_id=2VRGKAVVXU3LRNTZZUCYLUFYSX2YCA7MAER33GCABLC4EFZAKP3A, seg_pos={0 0}, upload=133198]: Done uploading piece. err: <nil> Nov 5 18:49:05.708723252 (151h8m37.404571995s) [noise=true , address=77.90.31.12:4590 , piece_id=KF5M57SIYYH5WBQZZ3KXQINWBYIIFA7ZVP6QWJORHMKZFFDCJ3MQ, seg_pos={0 0}, upload=133198]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: write tcp 188.166.225.170:57910->77.90.31.12:4590: use of closed network connection; write tcp 188.166.225.170:57910->77.90.31.12:4590: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57910->77.90.31.12:4590: use of closed network connection Nov 5 18:49:05.708984810 (151h8m37.404833563s) [noise=true , address=77.90.31.19:4842 , piece_id=XA47EB4PMODRDDVWIP2YJ2XYAGRHCMEDQTYFHQGUCJCOPPI6OT2Q, seg_pos={0 0}, upload=133198]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: read tcp 188.166.225.170:35846->77.90.31.19:4842: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 18:49:05.709477037 (151h8m37.4053258s) [noise=true , address=77.90.31.14:4663 , piece_id=DZJYBZ2I24SBIKJNKVI5TPGDWMHGYKMNPCDOIDDM2MSPWIH4OYVQ, seg_pos={0 0}, upload=133198]: Done uploading piece. err: ecclient: upload failed (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, address:77.90.31.14:4663): protocol: read tcp 188.166.225.170:55070->77.90.31.14:4663: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 18:49:05.709661409 (151h8m37.405510162s) [noise=true , address=45.91.116.6:4974 , piece_id=VIZKH56GJABCBBBTA3W5CGSUS645JINBDPVR5YBTWOGQ77I5YZ4Q, seg_pos={0 0}, upload=133198]: Done uploading piece. err: ecclient: upload failed (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, address:45.91.116.6:4974): protocol: read tcp 188.166.225.170:41618->45.91.116.6:4974: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 18:49:05.709692888 (151h8m37.405541642s) [seg_pos={0 0}, upload=133198]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 5 18:49:05.951688341 (151h8m37.647537094s) [upload=133198]: Done uploading segments... Nov 5 18:49:05.951732825 (151h8m37.647581568s) [upload=133198]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.912791396s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [........................................................XX..] duration: 3.47195414s trace: https://jaeger.dev.storj.io/trace/3ea7c69f954229ec FAIL (2025-11-05 18:50:43 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.472001169s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 9.296898156s trace: https://jaeger.dev.storj.io/trace/4578568744f6a3d6 FAIL (2025-11-05 18:50:32 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 5 18:48:36.399583130 (151h8m8.095431873s) [upload=133198]: Starting upload Nov 5 18:48:36.399692318 (151h8m8.095541071s) [upload=133198]: Done starting upload Nov 5 18:48:36.399752994 (151h8m8.095601737s) [upload=133198]: Uploading segments... Nov 5 18:48:36.399936253 (151h8m8.095785007s) [upload=133198]: Got next segment. Inline: false Nov 5 18:48:36.399987891 (151h8m8.095836644s) [upload=133198]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 5 18:48:36.399993442 (151h8m8.095842195s) [upload=133198]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 5 18:48:36.647833063 (151h8m8.343681807s) [seg_pos={0 0}, upload=133198]: Begin upload segment... Nov 5 18:48:36.647875144 (151h8m8.343723887s) [seg_pos={0 0}, upload=133198]: Done begin upload segment. Nov 5 18:48:36.647901734 (151h8m8.343750487s) [noise=true , address=45.91.116.6:4974 , piece_id=VIZKH56GJABCBBBTA3W5CGSUS645JINBDPVR5YBTWOGQ77I5YZ4Q, seg_pos={0 0}, upload=133198]: Uploading piece... Nov 5 18:48:36.649986123 (151h8m8.345834867s) [noise=true , address=145.79.227.23:4545 , piece_id=2VRGKAVVXU3LRNTZZUCYLUFYSX2YCA7MAER33GCABLC4EFZAKP3A, seg_pos={0 0}, upload=133198]: Uploading piece... Nov 5 18:48:36.650653094 (151h8m8.346501847s) [noise=true , address=145.79.227.17:4972 , piece_id=AGKRIHAOAC37VCKRUO2MKC4BUP34BC3DL6OJ56JBVPNDAQL4C2UQ, seg_pos={0 0}, upload=133198]: Uploading piece... Nov 5 18:48:36.651122347 (151h8m8.3469711s) [noise=true , address=145.79.227.25:4618 , piece_id=C5GZJZMGBHWQTIPOYBLZTQPNHTKIEIN2TFQYZOH73QI3V6UKQMFQ, seg_pos={0 0}, upload=133198]: Uploading piece... Nov 5 18:48:36.651439301 (151h8m8.347288044s) [noise=true , address=77.90.31.14:4663 , piece_id=DZJYBZ2I24SBIKJNKVI5TPGDWMHGYKMNPCDOIDDM2MSPWIH4OYVQ, seg_pos={0 0}, upload=133198]: Uploading piece... Nov 5 18:48:36.652593810 (151h8m8.348442563s) [noise=true , address=145.79.227.16:4911 , piece_id=CPHIIOPUWLUE6B2VHM3XVOJZAVXICEZF4DNMDCDB5FZEIOJH2T4Q, seg_pos={0 0}, upload=133198]: Uploading piece... Nov 5 18:48:36.653013098 (151h8m8.348861851s) [noise=true , address=77.90.31.19:4842 , piece_id=XA47EB4PMODRDDVWIP2YJ2XYAGRHCMEDQTYFHQGUCJCOPPI6OT2Q, seg_pos={0 0}, upload=133198]: Uploading piece... Nov 5 18:48:36.653698693 (151h8m8.349547446s) [noise=true , address=145.79.227.11:4729 , piece_id=GSVCIJUKZUDXLMJP6O7E347MSST3TW53LHJVH5ACVPS46EPIUVWA, seg_pos={0 0}, upload=133198]: Uploading piece... Nov 5 18:48:36.654497464 (151h8m8.350346217s) [noise=true , address=145.79.227.14:4838 , piece_id=NU3PLF3XK4MJQVYPK5ORBSR5Z7KBIKQB4GSPR77TTRGH6HWVNPIQ, seg_pos={0 0}, upload=133198]: Uploading piece... Nov 5 18:48:36.654918918 (151h8m8.350767661s) [noise=true , address=145.79.227.31:4799 , piece_id=WA2DSLH4AH6AHWZUOANQ4U2PWGSSU4F6DQNY5YIG3FYKWNQOWYCQ, seg_pos={0 0}, upload=133198]: Uploading piece... Nov 5 18:48:36.656003563 (151h8m8.351852316s) [noise=true , address=145.79.227.27:4690 , piece_id=YFCXBO7UPR3JND6PFUSUPAV32WU3YUXJPPN62UE4KIBZHCWGXMMQ, seg_pos={0 0}, upload=133198]: Uploading piece... Nov 5 18:48:36.656332689 (151h8m8.352181442s) [noise=true , address=145.79.227.9:4367 , piece_id=UND6AUULHZL67V2ZUVMTUHRXV3I2RFGYOEOINE6ERKUAQSC6AXYQ, seg_pos={0 0}, upload=133198]: Uploading piece... Nov 5 18:48:36.657419229 (151h8m8.353267972s) [noise=true , address=77.90.31.12:4590 , piece_id=KF5M57SIYYH5WBQZZ3KXQINWBYIIFA7ZVP6QWJORHMKZFFDCJ3MQ, seg_pos={0 0}, upload=133198]: Uploading piece... Nov 5 18:48:36.658763309 (151h8m8.354612052s) [noise=true , address=145.79.227.20:4437 , piece_id=PTYH2CNZUY6VSRDA5HDVQ5YCAJD26J5GIUM6KXRRK3DPWTOBFBKA, seg_pos={0 0}, upload=133198]: Uploading piece... Nov 5 18:48:36.661246016 (151h8m8.357094769s) [noise=true , address=145.79.227.8:4331 , piece_id=OZGLAEOKHMCUZX2CEOB5AJCN7K6AUDUGYRWJUTQZEEW26OZVUDOA, seg_pos={0 0}, upload=133198]: Uploading piece... Nov 5 18:48:36.662727748 (151h8m8.358576501s) [noise=true , address=145.79.227.32:4834 , piece_id=MQAE3ZGOCAS2QSKHBR4SJO2UVDVKRQPHILI5JR7NE46JG6UYPI5Q, seg_pos={0 0}, upload=133198]: Uploading piece... Nov 5 18:48:38.266459954 (151h8m9.962308697s) [upload=133198]: Next returned nil segment Nov 5 18:48:38.266474912 (151h8m9.962323665s) [upload=133198]: Waiting for error group managing segments... Nov 5 18:48:38.648038210 (151h8m10.343886963s) [noise=true , address=145.79.227.17:4972 , piece_id=AGKRIHAOAC37VCKRUO2MKC4BUP34BC3DL6OJ56JBVPNDAQL4C2UQ, seg_pos={0 0}, upload=133198]: Done uploading piece. err: <nil> Nov 5 18:48:38.649794854 (151h8m10.345643607s) [noise=true , address=145.79.227.11:4729 , piece_id=GSVCIJUKZUDXLMJP6O7E347MSST3TW53LHJVH5ACVPS46EPIUVWA, seg_pos={0 0}, upload=133198]: Done uploading piece. err: <nil> Nov 5 18:48:38.653624727 (151h8m10.34947348s) [noise=true , address=145.79.227.32:4834 , piece_id=MQAE3ZGOCAS2QSKHBR4SJO2UVDVKRQPHILI5JR7NE46JG6UYPI5Q, seg_pos={0 0}, upload=133198]: Done uploading piece. err: <nil> Nov 5 18:48:38.654714322 (151h8m10.350563075s) [noise=true , address=145.79.227.8:4331 , piece_id=OZGLAEOKHMCUZX2CEOB5AJCN7K6AUDUGYRWJUTQZEEW26OZVUDOA, seg_pos={0 0}, upload=133198]: Done uploading piece. err: <nil> Nov 5 18:48:38.655492113 (151h8m10.351340856s) [noise=true , address=145.79.227.14:4838 , piece_id=NU3PLF3XK4MJQVYPK5ORBSR5Z7KBIKQB4GSPR77TTRGH6HWVNPIQ, seg_pos={0 0}, upload=133198]: Done uploading piece. err: <nil> Nov 5 18:48:38.656244135 (151h8m10.352092888s) [noise=true , address=145.79.227.16:4911 , piece_id=CPHIIOPUWLUE6B2VHM3XVOJZAVXICEZF4DNMDCDB5FZEIOJH2T4Q, seg_pos={0 0}, upload=133198]: Done uploading piece. err: <nil> Nov 5 18:48:38.656809301 (151h8m10.352658054s) [noise=true , address=145.79.227.25:4618 , piece_id=C5GZJZMGBHWQTIPOYBLZTQPNHTKIEIN2TFQYZOH73QI3V6UKQMFQ, seg_pos={0 0}, upload=133198]: Done uploading piece. err: <nil> Nov 5 18:48:38.659785509 (151h8m10.355634262s) [noise=true , address=145.79.227.20:4437 , piece_id=PTYH2CNZUY6VSRDA5HDVQ5YCAJD26J5GIUM6KXRRK3DPWTOBFBKA, seg_pos={0 0}, upload=133198]: Done uploading piece. err: <nil> Nov 5 18:48:38.662194085 (151h8m10.358042848s) [noise=true , address=145.79.227.27:4690 , piece_id=YFCXBO7UPR3JND6PFUSUPAV32WU3YUXJPPN62UE4KIBZHCWGXMMQ, seg_pos={0 0}, upload=133198]: Done uploading piece. err: <nil> Nov 5 18:48:38.708392088 (151h8m10.404240841s) [noise=true , address=145.79.227.9:4367 , piece_id=UND6AUULHZL67V2ZUVMTUHRXV3I2RFGYOEOINE6ERKUAQSC6AXYQ, seg_pos={0 0}, upload=133198]: Done uploading piece. err: <nil> Nov 5 18:48:38.728515969 (151h8m10.424364712s) [noise=true , address=145.79.227.31:4799 , piece_id=WA2DSLH4AH6AHWZUOANQ4U2PWGSSU4F6DQNY5YIG3FYKWNQOWYCQ, seg_pos={0 0}, upload=133198]: Done uploading piece. err: <nil> Nov 5 18:48:38.738513017 (151h8m10.43436176s) [noise=true , address=145.79.227.23:4545 , piece_id=2VRGKAVVXU3LRNTZZUCYLUFYSX2YCA7MAER33GCABLC4EFZAKP3A, seg_pos={0 0}, upload=133198]: Done uploading piece. err: <nil> Nov 5 18:49:05.708723252 (151h8m37.404571995s) [noise=true , address=77.90.31.12:4590 , piece_id=KF5M57SIYYH5WBQZZ3KXQINWBYIIFA7ZVP6QWJORHMKZFFDCJ3MQ, seg_pos={0 0}, upload=133198]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: write tcp 188.166.225.170:57910->77.90.31.12:4590: use of closed network connection; write tcp 188.166.225.170:57910->77.90.31.12:4590: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57910->77.90.31.12:4590: use of closed network connection Nov 5 18:49:05.708984810 (151h8m37.404833563s) [noise=true , address=77.90.31.19:4842 , piece_id=XA47EB4PMODRDDVWIP2YJ2XYAGRHCMEDQTYFHQGUCJCOPPI6OT2Q, seg_pos={0 0}, upload=133198]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: read tcp 188.166.225.170:35846->77.90.31.19:4842: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 18:49:05.709477037 (151h8m37.4053258s) [noise=true , address=77.90.31.14:4663 , piece_id=DZJYBZ2I24SBIKJNKVI5TPGDWMHGYKMNPCDOIDDM2MSPWIH4OYVQ, seg_pos={0 0}, upload=133198]: Done uploading piece. err: ecclient: upload failed (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, address:77.90.31.14:4663): protocol: read tcp 188.166.225.170:55070->77.90.31.14:4663: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 18:49:05.709661409 (151h8m37.405510162s) [noise=true , address=45.91.116.6:4974 , piece_id=VIZKH56GJABCBBBTA3W5CGSUS645JINBDPVR5YBTWOGQ77I5YZ4Q, seg_pos={0 0}, upload=133198]: Done uploading piece. err: ecclient: upload failed (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, address:45.91.116.6:4974): protocol: read tcp 188.166.225.170:41618->45.91.116.6:4974: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 18:49:05.709692888 (151h8m37.405541642s) [seg_pos={0 0}, upload=133198]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 5 18:49:05.951688341 (151h8m37.647537094s) [upload=133198]: Done uploading segments... Nov 5 18:49:05.951732825 (151h8m37.647581568s) [upload=133198]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 9.29694266s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 5.768049815s trace: https://jaeger.dev.storj.io/trace/564bf2068e5ded38 FAIL (2025-11-05 18:50:29 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.768159614s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 6.262753126s trace: https://jaeger.dev.storj.io/trace/58e8c1d3b7ee8f0a FAIL (2025-11-05 18:50:19 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.262798212s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [.........................................................XX.] duration: 3.428246783s trace: https://jaeger.dev.storj.io/trace/3ea7c69f954229ec FAIL (2025-11-05 18:50:12 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.428277973s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 5.993344093s trace: https://jaeger.dev.storj.io/trace/4578568744f6a3d6 FAIL (2025-11-05 18:49:54 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 5 18:48:36.399583130 (151h8m8.095431873s) [upload=133198]: Starting upload Nov 5 18:48:36.399692318 (151h8m8.095541071s) [upload=133198]: Done starting upload Nov 5 18:48:36.399752994 (151h8m8.095601737s) [upload=133198]: Uploading segments... Nov 5 18:48:36.399936253 (151h8m8.095785007s) [upload=133198]: Got next segment. Inline: false Nov 5 18:48:36.399987891 (151h8m8.095836644s) [upload=133198]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 5 18:48:36.399993442 (151h8m8.095842195s) [upload=133198]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 5 18:48:36.647833063 (151h8m8.343681807s) [seg_pos={0 0}, upload=133198]: Begin upload segment... Nov 5 18:48:36.647875144 (151h8m8.343723887s) [seg_pos={0 0}, upload=133198]: Done begin upload segment. Nov 5 18:48:36.647901734 (151h8m8.343750487s) [noise=true , address=45.91.116.6:4974 , piece_id=VIZKH56GJABCBBBTA3W5CGSUS645JINBDPVR5YBTWOGQ77I5YZ4Q, seg_pos={0 0}, upload=133198]: Uploading piece... Nov 5 18:48:36.649986123 (151h8m8.345834867s) [noise=true , address=145.79.227.23:4545 , piece_id=2VRGKAVVXU3LRNTZZUCYLUFYSX2YCA7MAER33GCABLC4EFZAKP3A, seg_pos={0 0}, upload=133198]: Uploading piece... Nov 5 18:48:36.650653094 (151h8m8.346501847s) [noise=true , address=145.79.227.17:4972 , piece_id=AGKRIHAOAC37VCKRUO2MKC4BUP34BC3DL6OJ56JBVPNDAQL4C2UQ, seg_pos={0 0}, upload=133198]: Uploading piece... Nov 5 18:48:36.651122347 (151h8m8.3469711s) [noise=true , address=145.79.227.25:4618 , piece_id=C5GZJZMGBHWQTIPOYBLZTQPNHTKIEIN2TFQYZOH73QI3V6UKQMFQ, seg_pos={0 0}, upload=133198]: Uploading piece... Nov 5 18:48:36.651439301 (151h8m8.347288044s) [noise=true , address=77.90.31.14:4663 , piece_id=DZJYBZ2I24SBIKJNKVI5TPGDWMHGYKMNPCDOIDDM2MSPWIH4OYVQ, seg_pos={0 0}, upload=133198]: Uploading piece... Nov 5 18:48:36.652593810 (151h8m8.348442563s) [noise=true , address=145.79.227.16:4911 , piece_id=CPHIIOPUWLUE6B2VHM3XVOJZAVXICEZF4DNMDCDB5FZEIOJH2T4Q, seg_pos={0 0}, upload=133198]: Uploading piece... Nov 5 18:48:36.653013098 (151h8m8.348861851s) [noise=true , address=77.90.31.19:4842 , piece_id=XA47EB4PMODRDDVWIP2YJ2XYAGRHCMEDQTYFHQGUCJCOPPI6OT2Q, seg_pos={0 0}, upload=133198]: Uploading piece... Nov 5 18:48:36.653698693 (151h8m8.349547446s) [noise=true , address=145.79.227.11:4729 , piece_id=GSVCIJUKZUDXLMJP6O7E347MSST3TW53LHJVH5ACVPS46EPIUVWA, seg_pos={0 0}, upload=133198]: Uploading piece... Nov 5 18:48:36.654497464 (151h8m8.350346217s) [noise=true , address=145.79.227.14:4838 , piece_id=NU3PLF3XK4MJQVYPK5ORBSR5Z7KBIKQB4GSPR77TTRGH6HWVNPIQ, seg_pos={0 0}, upload=133198]: Uploading piece... Nov 5 18:48:36.654918918 (151h8m8.350767661s) [noise=true , address=145.79.227.31:4799 , piece_id=WA2DSLH4AH6AHWZUOANQ4U2PWGSSU4F6DQNY5YIG3FYKWNQOWYCQ, seg_pos={0 0}, upload=133198]: Uploading piece... Nov 5 18:48:36.656003563 (151h8m8.351852316s) [noise=true , address=145.79.227.27:4690 , piece_id=YFCXBO7UPR3JND6PFUSUPAV32WU3YUXJPPN62UE4KIBZHCWGXMMQ, seg_pos={0 0}, upload=133198]: Uploading piece... Nov 5 18:48:36.656332689 (151h8m8.352181442s) [noise=true , address=145.79.227.9:4367 , piece_id=UND6AUULHZL67V2ZUVMTUHRXV3I2RFGYOEOINE6ERKUAQSC6AXYQ, seg_pos={0 0}, upload=133198]: Uploading piece... Nov 5 18:48:36.657419229 (151h8m8.353267972s) [noise=true , address=77.90.31.12:4590 , piece_id=KF5M57SIYYH5WBQZZ3KXQINWBYIIFA7ZVP6QWJORHMKZFFDCJ3MQ, seg_pos={0 0}, upload=133198]: Uploading piece... Nov 5 18:48:36.658763309 (151h8m8.354612052s) [noise=true , address=145.79.227.20:4437 , piece_id=PTYH2CNZUY6VSRDA5HDVQ5YCAJD26J5GIUM6KXRRK3DPWTOBFBKA, seg_pos={0 0}, upload=133198]: Uploading piece... Nov 5 18:48:36.661246016 (151h8m8.357094769s) [noise=true , address=145.79.227.8:4331 , piece_id=OZGLAEOKHMCUZX2CEOB5AJCN7K6AUDUGYRWJUTQZEEW26OZVUDOA, seg_pos={0 0}, upload=133198]: Uploading piece... Nov 5 18:48:36.662727748 (151h8m8.358576501s) [noise=true , address=145.79.227.32:4834 , piece_id=MQAE3ZGOCAS2QSKHBR4SJO2UVDVKRQPHILI5JR7NE46JG6UYPI5Q, seg_pos={0 0}, upload=133198]: Uploading piece... Nov 5 18:48:38.266459954 (151h8m9.962308697s) [upload=133198]: Next returned nil segment Nov 5 18:48:38.266474912 (151h8m9.962323665s) [upload=133198]: Waiting for error group managing segments... Nov 5 18:48:38.648038210 (151h8m10.343886963s) [noise=true , address=145.79.227.17:4972 , piece_id=AGKRIHAOAC37VCKRUO2MKC4BUP34BC3DL6OJ56JBVPNDAQL4C2UQ, seg_pos={0 0}, upload=133198]: Done uploading piece. err: <nil> Nov 5 18:48:38.649794854 (151h8m10.345643607s) [noise=true , address=145.79.227.11:4729 , piece_id=GSVCIJUKZUDXLMJP6O7E347MSST3TW53LHJVH5ACVPS46EPIUVWA, seg_pos={0 0}, upload=133198]: Done uploading piece. err: <nil> Nov 5 18:48:38.653624727 (151h8m10.34947348s) [noise=true , address=145.79.227.32:4834 , piece_id=MQAE3ZGOCAS2QSKHBR4SJO2UVDVKRQPHILI5JR7NE46JG6UYPI5Q, seg_pos={0 0}, upload=133198]: Done uploading piece. err: <nil> Nov 5 18:48:38.654714322 (151h8m10.350563075s) [noise=true , address=145.79.227.8:4331 , piece_id=OZGLAEOKHMCUZX2CEOB5AJCN7K6AUDUGYRWJUTQZEEW26OZVUDOA, seg_pos={0 0}, upload=133198]: Done uploading piece. err: <nil> Nov 5 18:48:38.655492113 (151h8m10.351340856s) [noise=true , address=145.79.227.14:4838 , piece_id=NU3PLF3XK4MJQVYPK5ORBSR5Z7KBIKQB4GSPR77TTRGH6HWVNPIQ, seg_pos={0 0}, upload=133198]: Done uploading piece. err: <nil> Nov 5 18:48:38.656244135 (151h8m10.352092888s) [noise=true , address=145.79.227.16:4911 , piece_id=CPHIIOPUWLUE6B2VHM3XVOJZAVXICEZF4DNMDCDB5FZEIOJH2T4Q, seg_pos={0 0}, upload=133198]: Done uploading piece. err: <nil> Nov 5 18:48:38.656809301 (151h8m10.352658054s) [noise=true , address=145.79.227.25:4618 , piece_id=C5GZJZMGBHWQTIPOYBLZTQPNHTKIEIN2TFQYZOH73QI3V6UKQMFQ, seg_pos={0 0}, upload=133198]: Done uploading piece. err: <nil> Nov 5 18:48:38.659785509 (151h8m10.355634262s) [noise=true , address=145.79.227.20:4437 , piece_id=PTYH2CNZUY6VSRDA5HDVQ5YCAJD26J5GIUM6KXRRK3DPWTOBFBKA, seg_pos={0 0}, upload=133198]: Done uploading piece. err: <nil> Nov 5 18:48:38.662194085 (151h8m10.358042848s) [noise=true , address=145.79.227.27:4690 , piece_id=YFCXBO7UPR3JND6PFUSUPAV32WU3YUXJPPN62UE4KIBZHCWGXMMQ, seg_pos={0 0}, upload=133198]: Done uploading piece. err: <nil> Nov 5 18:48:38.708392088 (151h8m10.404240841s) [noise=true , address=145.79.227.9:4367 , piece_id=UND6AUULHZL67V2ZUVMTUHRXV3I2RFGYOEOINE6ERKUAQSC6AXYQ, seg_pos={0 0}, upload=133198]: Done uploading piece. err: <nil> Nov 5 18:48:38.728515969 (151h8m10.424364712s) [noise=true , address=145.79.227.31:4799 , piece_id=WA2DSLH4AH6AHWZUOANQ4U2PWGSSU4F6DQNY5YIG3FYKWNQOWYCQ, seg_pos={0 0}, upload=133198]: Done uploading piece. err: <nil> Nov 5 18:48:38.738513017 (151h8m10.43436176s) [noise=true , address=145.79.227.23:4545 , piece_id=2VRGKAVVXU3LRNTZZUCYLUFYSX2YCA7MAER33GCABLC4EFZAKP3A, seg_pos={0 0}, upload=133198]: Done uploading piece. err: <nil> Nov 5 18:49:05.708723252 (151h8m37.404571995s) [noise=true , address=77.90.31.12:4590 , piece_id=KF5M57SIYYH5WBQZZ3KXQINWBYIIFA7ZVP6QWJORHMKZFFDCJ3MQ, seg_pos={0 0}, upload=133198]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: write tcp 188.166.225.170:57910->77.90.31.12:4590: use of closed network connection; write tcp 188.166.225.170:57910->77.90.31.12:4590: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57910->77.90.31.12:4590: use of closed network connection Nov 5 18:49:05.708984810 (151h8m37.404833563s) [noise=true , address=77.90.31.19:4842 , piece_id=XA47EB4PMODRDDVWIP2YJ2XYAGRHCMEDQTYFHQGUCJCOPPI6OT2Q, seg_pos={0 0}, upload=133198]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: read tcp 188.166.225.170:35846->77.90.31.19:4842: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 18:49:05.709477037 (151h8m37.4053258s) [noise=true , address=77.90.31.14:4663 , piece_id=DZJYBZ2I24SBIKJNKVI5TPGDWMHGYKMNPCDOIDDM2MSPWIH4OYVQ, seg_pos={0 0}, upload=133198]: Done uploading piece. err: ecclient: upload failed (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, address:77.90.31.14:4663): protocol: read tcp 188.166.225.170:55070->77.90.31.14:4663: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 18:49:05.709661409 (151h8m37.405510162s) [noise=true , address=45.91.116.6:4974 , piece_id=VIZKH56GJABCBBBTA3W5CGSUS645JINBDPVR5YBTWOGQ77I5YZ4Q, seg_pos={0 0}, upload=133198]: Done uploading piece. err: ecclient: upload failed (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, address:45.91.116.6:4974): protocol: read tcp 188.166.225.170:41618->45.91.116.6:4974: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 18:49:05.709692888 (151h8m37.405541642s) [seg_pos={0 0}, upload=133198]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 5 18:49:05.951688341 (151h8m37.647537094s) [upload=133198]: Done uploading segments... Nov 5 18:49:05.951732825 (151h8m37.647581568s) [upload=133198]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.993374402s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 6.468440509s trace: https://jaeger.dev.storj.io/trace/58e8c1d3b7ee8f0a FAIL (2025-11-05 18:49:27 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.468495444s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 6.969244079s trace: https://jaeger.dev.storj.io/trace/564bf2068e5ded38 FAIL (2025-11-05 18:49:24 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.969259228s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 1m0.003322817s trace: https://jaeger.dev.storj.io/trace/564bf2068e5ded38 FAIL (2025-11-05 18:49:17 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.00334956s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 30.244916874s trace: https://jaeger.dev.storj.io/trace/4578568744f6a3d6 FAIL (2025-11-05 18:49:05 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 5 18:48:36.399583130 (151h8m8.095431873s) [upload=133198]: Starting upload Nov 5 18:48:36.399692318 (151h8m8.095541071s) [upload=133198]: Done starting upload Nov 5 18:48:36.399752994 (151h8m8.095601737s) [upload=133198]: Uploading segments... Nov 5 18:48:36.399936253 (151h8m8.095785007s) [upload=133198]: Got next segment. Inline: false Nov 5 18:48:36.399987891 (151h8m8.095836644s) [upload=133198]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 5 18:48:36.399993442 (151h8m8.095842195s) [upload=133198]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 5 18:48:36.647833063 (151h8m8.343681807s) [seg_pos={0 0}, upload=133198]: Begin upload segment... Nov 5 18:48:36.647875144 (151h8m8.343723887s) [seg_pos={0 0}, upload=133198]: Done begin upload segment. Nov 5 18:48:36.647901734 (151h8m8.343750487s) [noise=true , address=45.91.116.6:4974 , piece_id=VIZKH56GJABCBBBTA3W5CGSUS645JINBDPVR5YBTWOGQ77I5YZ4Q, seg_pos={0 0}, upload=133198]: Uploading piece... Nov 5 18:48:36.649986123 (151h8m8.345834867s) [noise=true , address=145.79.227.23:4545 , piece_id=2VRGKAVVXU3LRNTZZUCYLUFYSX2YCA7MAER33GCABLC4EFZAKP3A, seg_pos={0 0}, upload=133198]: Uploading piece... Nov 5 18:48:36.650653094 (151h8m8.346501847s) [noise=true , address=145.79.227.17:4972 , piece_id=AGKRIHAOAC37VCKRUO2MKC4BUP34BC3DL6OJ56JBVPNDAQL4C2UQ, seg_pos={0 0}, upload=133198]: Uploading piece... Nov 5 18:48:36.651122347 (151h8m8.3469711s) [noise=true , address=145.79.227.25:4618 , piece_id=C5GZJZMGBHWQTIPOYBLZTQPNHTKIEIN2TFQYZOH73QI3V6UKQMFQ, seg_pos={0 0}, upload=133198]: Uploading piece... Nov 5 18:48:36.651439301 (151h8m8.347288044s) [noise=true , address=77.90.31.14:4663 , piece_id=DZJYBZ2I24SBIKJNKVI5TPGDWMHGYKMNPCDOIDDM2MSPWIH4OYVQ, seg_pos={0 0}, upload=133198]: Uploading piece... Nov 5 18:48:36.652593810 (151h8m8.348442563s) [noise=true , address=145.79.227.16:4911 , piece_id=CPHIIOPUWLUE6B2VHM3XVOJZAVXICEZF4DNMDCDB5FZEIOJH2T4Q, seg_pos={0 0}, upload=133198]: Uploading piece... Nov 5 18:48:36.653013098 (151h8m8.348861851s) [noise=true , address=77.90.31.19:4842 , piece_id=XA47EB4PMODRDDVWIP2YJ2XYAGRHCMEDQTYFHQGUCJCOPPI6OT2Q, seg_pos={0 0}, upload=133198]: Uploading piece... Nov 5 18:48:36.653698693 (151h8m8.349547446s) [noise=true , address=145.79.227.11:4729 , piece_id=GSVCIJUKZUDXLMJP6O7E347MSST3TW53LHJVH5ACVPS46EPIUVWA, seg_pos={0 0}, upload=133198]: Uploading piece... Nov 5 18:48:36.654497464 (151h8m8.350346217s) [noise=true , address=145.79.227.14:4838 , piece_id=NU3PLF3XK4MJQVYPK5ORBSR5Z7KBIKQB4GSPR77TTRGH6HWVNPIQ, seg_pos={0 0}, upload=133198]: Uploading piece... Nov 5 18:48:36.654918918 (151h8m8.350767661s) [noise=true , address=145.79.227.31:4799 , piece_id=WA2DSLH4AH6AHWZUOANQ4U2PWGSSU4F6DQNY5YIG3FYKWNQOWYCQ, seg_pos={0 0}, upload=133198]: Uploading piece... Nov 5 18:48:36.656003563 (151h8m8.351852316s) [noise=true , address=145.79.227.27:4690 , piece_id=YFCXBO7UPR3JND6PFUSUPAV32WU3YUXJPPN62UE4KIBZHCWGXMMQ, seg_pos={0 0}, upload=133198]: Uploading piece... Nov 5 18:48:36.656332689 (151h8m8.352181442s) [noise=true , address=145.79.227.9:4367 , piece_id=UND6AUULHZL67V2ZUVMTUHRXV3I2RFGYOEOINE6ERKUAQSC6AXYQ, seg_pos={0 0}, upload=133198]: Uploading piece... Nov 5 18:48:36.657419229 (151h8m8.353267972s) [noise=true , address=77.90.31.12:4590 , piece_id=KF5M57SIYYH5WBQZZ3KXQINWBYIIFA7ZVP6QWJORHMKZFFDCJ3MQ, seg_pos={0 0}, upload=133198]: Uploading piece... Nov 5 18:48:36.658763309 (151h8m8.354612052s) [noise=true , address=145.79.227.20:4437 , piece_id=PTYH2CNZUY6VSRDA5HDVQ5YCAJD26J5GIUM6KXRRK3DPWTOBFBKA, seg_pos={0 0}, upload=133198]: Uploading piece... Nov 5 18:48:36.661246016 (151h8m8.357094769s) [noise=true , address=145.79.227.8:4331 , piece_id=OZGLAEOKHMCUZX2CEOB5AJCN7K6AUDUGYRWJUTQZEEW26OZVUDOA, seg_pos={0 0}, upload=133198]: Uploading piece... Nov 5 18:48:36.662727748 (151h8m8.358576501s) [noise=true , address=145.79.227.32:4834 , piece_id=MQAE3ZGOCAS2QSKHBR4SJO2UVDVKRQPHILI5JR7NE46JG6UYPI5Q, seg_pos={0 0}, upload=133198]: Uploading piece... Nov 5 18:48:38.266459954 (151h8m9.962308697s) [upload=133198]: Next returned nil segment Nov 5 18:48:38.266474912 (151h8m9.962323665s) [upload=133198]: Waiting for error group managing segments... Nov 5 18:48:38.648038210 (151h8m10.343886963s) [noise=true , address=145.79.227.17:4972 , piece_id=AGKRIHAOAC37VCKRUO2MKC4BUP34BC3DL6OJ56JBVPNDAQL4C2UQ, seg_pos={0 0}, upload=133198]: Done uploading piece. err: <nil> Nov 5 18:48:38.649794854 (151h8m10.345643607s) [noise=true , address=145.79.227.11:4729 , piece_id=GSVCIJUKZUDXLMJP6O7E347MSST3TW53LHJVH5ACVPS46EPIUVWA, seg_pos={0 0}, upload=133198]: Done uploading piece. err: <nil> Nov 5 18:48:38.653624727 (151h8m10.34947348s) [noise=true , address=145.79.227.32:4834 , piece_id=MQAE3ZGOCAS2QSKHBR4SJO2UVDVKRQPHILI5JR7NE46JG6UYPI5Q, seg_pos={0 0}, upload=133198]: Done uploading piece. err: <nil> Nov 5 18:48:38.654714322 (151h8m10.350563075s) [noise=true , address=145.79.227.8:4331 , piece_id=OZGLAEOKHMCUZX2CEOB5AJCN7K6AUDUGYRWJUTQZEEW26OZVUDOA, seg_pos={0 0}, upload=133198]: Done uploading piece. err: <nil> Nov 5 18:48:38.655492113 (151h8m10.351340856s) [noise=true , address=145.79.227.14:4838 , piece_id=NU3PLF3XK4MJQVYPK5ORBSR5Z7KBIKQB4GSPR77TTRGH6HWVNPIQ, seg_pos={0 0}, upload=133198]: Done uploading piece. err: <nil> Nov 5 18:48:38.656244135 (151h8m10.352092888s) [noise=true , address=145.79.227.16:4911 , piece_id=CPHIIOPUWLUE6B2VHM3XVOJZAVXICEZF4DNMDCDB5FZEIOJH2T4Q, seg_pos={0 0}, upload=133198]: Done uploading piece. err: <nil> Nov 5 18:48:38.656809301 (151h8m10.352658054s) [noise=true , address=145.79.227.25:4618 , piece_id=C5GZJZMGBHWQTIPOYBLZTQPNHTKIEIN2TFQYZOH73QI3V6UKQMFQ, seg_pos={0 0}, upload=133198]: Done uploading piece. err: <nil> Nov 5 18:48:38.659785509 (151h8m10.355634262s) [noise=true , address=145.79.227.20:4437 , piece_id=PTYH2CNZUY6VSRDA5HDVQ5YCAJD26J5GIUM6KXRRK3DPWTOBFBKA, seg_pos={0 0}, upload=133198]: Done uploading piece. err: <nil> Nov 5 18:48:38.662194085 (151h8m10.358042848s) [noise=true , address=145.79.227.27:4690 , piece_id=YFCXBO7UPR3JND6PFUSUPAV32WU3YUXJPPN62UE4KIBZHCWGXMMQ, seg_pos={0 0}, upload=133198]: Done uploading piece. err: <nil> Nov 5 18:48:38.708392088 (151h8m10.404240841s) [noise=true , address=145.79.227.9:4367 , piece_id=UND6AUULHZL67V2ZUVMTUHRXV3I2RFGYOEOINE6ERKUAQSC6AXYQ, seg_pos={0 0}, upload=133198]: Done uploading piece. err: <nil> Nov 5 18:48:38.728515969 (151h8m10.424364712s) [noise=true , address=145.79.227.31:4799 , piece_id=WA2DSLH4AH6AHWZUOANQ4U2PWGSSU4F6DQNY5YIG3FYKWNQOWYCQ, seg_pos={0 0}, upload=133198]: Done uploading piece. err: <nil> Nov 5 18:48:38.738513017 (151h8m10.43436176s) [noise=true , address=145.79.227.23:4545 , piece_id=2VRGKAVVXU3LRNTZZUCYLUFYSX2YCA7MAER33GCABLC4EFZAKP3A, seg_pos={0 0}, upload=133198]: Done uploading piece. err: <nil> Nov 5 18:49:05.708723252 (151h8m37.404571995s) [noise=true , address=77.90.31.12:4590 , piece_id=KF5M57SIYYH5WBQZZ3KXQINWBYIIFA7ZVP6QWJORHMKZFFDCJ3MQ, seg_pos={0 0}, upload=133198]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: write tcp 188.166.225.170:57910->77.90.31.12:4590: use of closed network connection; write tcp 188.166.225.170:57910->77.90.31.12:4590: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57910->77.90.31.12:4590: use of closed network connection Nov 5 18:49:05.708984810 (151h8m37.404833563s) [noise=true , address=77.90.31.19:4842 , piece_id=XA47EB4PMODRDDVWIP2YJ2XYAGRHCMEDQTYFHQGUCJCOPPI6OT2Q, seg_pos={0 0}, upload=133198]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: read tcp 188.166.225.170:35846->77.90.31.19:4842: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 18:49:05.709477037 (151h8m37.4053258s) [noise=true , address=77.90.31.14:4663 , piece_id=DZJYBZ2I24SBIKJNKVI5TPGDWMHGYKMNPCDOIDDM2MSPWIH4OYVQ, seg_pos={0 0}, upload=133198]: Done uploading piece. err: ecclient: upload failed (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, address:77.90.31.14:4663): protocol: read tcp 188.166.225.170:55070->77.90.31.14:4663: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 18:49:05.709661409 (151h8m37.405510162s) [noise=true , address=45.91.116.6:4974 , piece_id=VIZKH56GJABCBBBTA3W5CGSUS645JINBDPVR5YBTWOGQ77I5YZ4Q, seg_pos={0 0}, upload=133198]: Done uploading piece. err: ecclient: upload failed (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, address:45.91.116.6:4974): protocol: read tcp 188.166.225.170:41618->45.91.116.6:4974: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 5 18:49:05.709692888 (151h8m37.405541642s) [seg_pos={0 0}, upload=133198]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 5 18:49:05.951688341 (151h8m37.647537094s) [upload=133198]: Done uploading segments... Nov 5 18:49:05.951732825 (151h8m37.647581568s) [upload=133198]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 30.244932185s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [..........................................................XX] duration: 5.000717925s trace: https://jaeger.dev.storj.io/trace/3ea7c69f954229ec FAIL (2025-11-05 18:48:59 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000782567s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 30.001036379s trace: https://jaeger.dev.storj.io/trace/58e8c1d3b7ee8f0a FAIL (2025-11-05 18:48:39 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.001089239s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...........................................................X] duration: 5.001383193s trace: https://jaeger.dev.storj.io/trace/7e5f3e38cd77448f FAIL (2025-11-05 18:48:28 UTC) orbiter warning: context deadline exceeded Duration: 5.001405666s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [X...........................................................] duration: 3.467815986s trace: https://jaeger.dev.storj.io/trace/2d7205061b923206 FAIL (2025-11-05 14:58:15 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.467841697s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.X..........................................................] duration: 3.476260371s trace: https://jaeger.dev.storj.io/trace/2d7205061b923206 FAIL (2025-11-05 14:57:13 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.476290127s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..X.........................................................] duration: 3.459361091s trace: https://jaeger.dev.storj.io/trace/2d7205061b923206 FAIL (2025-11-05 14:56:12 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.459399124s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...X........................................................] duration: 3.397255867s trace: https://jaeger.dev.storj.io/trace/2d7205061b923206 FAIL (2025-11-05 14:55:52 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.397317876s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [....X.......................................................] duration: 3.411712004s trace: https://jaeger.dev.storj.io/trace/2d7205061b923206 FAIL (2025-11-05 14:55:29 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.411766538s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.....X......................................................] duration: 3.416097344s trace: https://jaeger.dev.storj.io/trace/2d7205061b923206 FAIL (2025-11-05 14:54:45 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.416116471s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [......X.....................................................] duration: 4.350291868s trace: https://jaeger.dev.storj.io/trace/2d7205061b923206 FAIL (2025-11-05 14:54:17 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.350322835s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.......X....................................................] duration: 3.449734175s trace: https://jaeger.dev.storj.io/trace/2d7205061b923206 FAIL (2025-11-05 14:53:08 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.44976799s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [........X...................................................] duration: 3.45505592s trace: https://jaeger.dev.storj.io/trace/2d7205061b923206 FAIL (2025-11-05 14:52:41 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.455128999s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.........X..................................................] duration: 3.514974968s trace: https://jaeger.dev.storj.io/trace/2d7205061b923206 FAIL (2025-11-05 14:51:36 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.515004765s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..........X.................................................] duration: 3.510337127s trace: https://jaeger.dev.storj.io/trace/2d7205061b923206 FAIL (2025-11-05 14:51:09 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.510370853s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...........X................................................] duration: 3.503359534s trace: https://jaeger.dev.storj.io/trace/2d7205061b923206 FAIL (2025-11-05 14:50:00 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.503387819s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [............X...............................................] duration: 3.374573397s trace: https://jaeger.dev.storj.io/trace/2d7205061b923206 FAIL (2025-11-05 14:49:33 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.374607072s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.............X..............................................] duration: 3.435725701s trace: https://jaeger.dev.storj.io/trace/2d7205061b923206 FAIL (2025-11-05 14:49:00 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.435754345s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..............X.............................................] duration: 3.421091054s trace: https://jaeger.dev.storj.io/trace/2d7205061b923206 FAIL (2025-11-05 14:47:52 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.421121582s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...............X............................................] duration: 3.45992767s trace: https://jaeger.dev.storj.io/trace/2d7205061b923206 FAIL (2025-11-05 14:47:16 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.459979558s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [................X...........................................] duration: 3.414734843s trace: https://jaeger.dev.storj.io/trace/2d7205061b923206 FAIL (2025-11-05 14:46:51 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.414769588s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.................X..........................................] duration: 3.43557863s trace: https://jaeger.dev.storj.io/trace/2d7205061b923206 FAIL (2025-11-05 14:46:24 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.435607155s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..................X.........................................] duration: 3.413376805s trace: https://jaeger.dev.storj.io/trace/2d7205061b923206 FAIL (2025-11-05 14:45:58 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.413395461s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...................X........................................] duration: 3.481726702s trace: https://jaeger.dev.storj.io/trace/2d7205061b923206 FAIL (2025-11-05 14:45:19 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.481756006s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [....................X.......................................] duration: 3.404485723s trace: https://jaeger.dev.storj.io/trace/2d7205061b923206 FAIL (2025-11-05 14:44:26 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.404513997s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.....................X......................................] duration: 3.439468688s trace: https://jaeger.dev.storj.io/trace/2d7205061b923206 FAIL (2025-11-05 14:43:56 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.439498885s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)