Orbiter History

checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [...............X..X.........................................] duration: 5.701032341s trace: https://jaeger.dev.storj.io/trace/3e2ae8884260b157 FAIL (2025-11-05 19:21:40 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.70105858s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...............X...X........................................] duration: 6.382739743s trace: https://jaeger.dev.storj.io/trace/74cf1e30f6d49950 FAIL (2025-11-05 19:21:28 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.382768067s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [.................XX...X.....................................] duration: 3.441885704s trace: https://jaeger.dev.storj.io/trace/204ab2570dc82d9c FAIL (2025-11-05 19:21:13 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.441933615s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [................X...X.......................................] duration: 6.189992077s trace: https://jaeger.dev.storj.io/trace/74cf1e30f6d49950 FAIL (2025-11-05 19:21:09 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.190023547s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..............X...X.........................................] duration: 6.005558124s trace: https://jaeger.dev.storj.io/trace/2e576aae48e8f03e FAIL (2025-11-05 19:21:08 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== 5 18:52:06.850167887 (151h11m38.54601663s) [noise=true , address=145.79.227.25:4618 , piece_id=XRUPY5BUPG7BXN2MI6MGYO4N3GW6TZDJE46MZFNRELWANHHG3N5A, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.850747301 (151h11m38.546596054s) [noise=true , address=77.90.31.15:4698 , piece_id=7VACL26N7ZCBXIA2I3XL5Q3UJ2BAHDIPXX4VZCQX3S63ABNW625Q, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.851222605 (151h11m38.547071358s) [noise=true , address=145.79.227.17:4972 , piece_id=H7UHQV7RDLCZUHY2XMXUAD2PX4ZBLWKTM47N6PH7UOGBRXPMWVOA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.851558575 (151h11m38.547407319s) [noise=true , address=145.79.227.22:4509 , piece_id=7RA5NJ72PI3DT4IC2ECVLZ3A2HXXL546ELVDXHZW4OXNN4D36OEA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.851909263 (151h11m38.547758006s) [noise=true , address=145.79.227.26:4654 , piece_id=E5J22KNPQSETSY443QQZXFFKUUJFHWFV22SQAVZ4BFQ4M6YWCNQQ, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.852327540 (151h11m38.548176293s) [noise=true , address=45.91.116.7:4010 , piece_id=EBVGGNQF3N4MTUIATJUWK4QZN7SEOWI6SIAW6N7JZOB5G6H33NEQ, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.852728624 (151h11m38.548577377s) [noise=true , address=145.79.227.9:4366 , piece_id=7ZRECKBQC4S6QLERBZY442WYS4H35XWOZYSGGPHSU7NQ6WL4UNVA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.853978774 (151h11m38.549827528s) [noise=true , address=77.90.31.18:4805 , piece_id=U2U34KK4AJK4A5HGNTHEYCIARLAR4X5NLOVCF63TN3PJUIRMMIHA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.854447296 (151h11m38.550296059s) [noise=true , address=77.90.31.16:4734 , piece_id=HFWVFM46PZFLZT6YNZVIA6HKLVN2PTV2GETZEKX356DMDGQO4HOA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.856523089 (151h11m38.552371842s) [noise=true , address=45.91.116.11:4117 , piece_id=QUGUYHEJWEEV733CBC3ONMV7XWDKVH2OFLQWXPQHPKZ7GTH4RLUA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.859490379 (151h11m38.555339122s) [noise=true , address=77.90.31.13:4628 , piece_id=QFTEHZ5FY7A5DPZZ44TRXGFRHPRNJZJHLQZDGRGOWYFEZYIZT2NQ, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.861350291 (151h11m38.557199034s) [noise=true , address=145.79.227.31:4797 , piece_id=2TFSZNURIAU573QUA5DUS7AJZNY2OXC6TJL3ST4VXIWY2WVB5YJA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.863368665 (151h11m38.559217418s) [noise=true , address=145.79.227.23:4547 , piece_id=K54H437KW5AAYUWOTMDBDP5GJ3ZABFKLAAOSS6WS5G5CROYEHODQ, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.864848794 (151h11m38.560697547s) [noise=true , address=145.79.227.24:4581 , piece_id=TSCVCH2KJEVMD4AVOEIHGTMRCRZ6NZMZA2CTQ6FUO2EIL3JAVYEA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.865644930 (151h11m38.561493683s) [noise=true , address=145.79.227.8:4330 , piece_id=4A47HIUX2MTJ2673XZZCMMF4T235TS4JWMNYJEVHHCKAHQX6APMA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:08.560504137 (151h11m40.25635289s) [upload=133242]: Next returned nil segment Nov 5 18:52:08.560522693 (151h11m40.256371446s) [upload=133242]: Waiting for error group managing segments... Nov 5 18:52:08.850333179 (151h11m40.546181932s) [noise=true , address=145.79.227.17:4972 , piece_id=H7UHQV7RDLCZUHY2XMXUAD2PX4ZBLWKTM47N6PH7UOGBRXPMWVOA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.853517822 (151h11m40.549366575s) [noise=true , address=145.79.227.22:4509 , piece_id=7RA5NJ72PI3DT4IC2ECVLZ3A2HXXL546ELVDXHZW4OXNN4D36OEA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.855547508 (151h11m40.551396261s) [noise=true , address=145.79.227.9:4366 , piece_id=7ZRECKBQC4S6QLERBZY442WYS4H35XWOZYSGGPHSU7NQ6WL4UNVA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.856103106 (151h11m40.551951859s) [noise=true , address=145.79.227.25:4618 , piece_id=XRUPY5BUPG7BXN2MI6MGYO4N3GW6TZDJE46MZFNRELWANHHG3N5A, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.860417321 (151h11m40.556266064s) [noise=true , address=145.79.227.8:4330 , piece_id=4A47HIUX2MTJ2673XZZCMMF4T235TS4JWMNYJEVHHCKAHQX6APMA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.865602796 (151h11m40.561451549s) [noise=true , address=145.79.227.23:4547 , piece_id=K54H437KW5AAYUWOTMDBDP5GJ3ZABFKLAAOSS6WS5G5CROYEHODQ, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.879393757 (151h11m40.57524251s) [noise=true , address=145.79.227.31:4797 , piece_id=2TFSZNURIAU573QUA5DUS7AJZNY2OXC6TJL3ST4VXIWY2WVB5YJA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.905541449 (151h11m40.601390202s) [noise=true , address=145.79.227.26:4654 , piece_id=E5J22KNPQSETSY443QQZXFFKUUJFHWFV22SQAVZ4BFQ4M6YWCNQQ, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.929018836 (151h11m40.624867589s) [noise=true , address=145.79.227.24:4581 , piece_id=TSCVCH2KJEVMD4AVOEIHGTMRCRZ6NZMZA2CTQ6FUO2EIL3JAVYEA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:35.899554917 (151h12m7.59540368s) [noise=true , address=77.90.31.17:4771 , piece_id=KVFE7VYW3RZJ25BWSH4BIRRQCMWDYOIK7EJ62XVREGN26YN7XYMQ, seg_pos={0 0}, upload=133242]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: write tcp 188.166.225.170:60676->77.90.31.17:4771: use of closed network connection; write tcp 188.166.225.170:60676->77.90.31.17:4771: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60676->77.90.31.17:4771: use of closed network connection Nov 5 18:52:35.899727506 (151h12m7.595576259s) [noise=true , address=77.90.31.13:4628 , piece_id=QFTEHZ5FY7A5DPZZ44TRXGFRHPRNJZJHLQZDGRGOWYFEZYIZT2NQ, seg_pos={0 0}, upload=133242]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: write tcp 188.166.225.170:42026->77.90.31.13:4628: use of closed network connection; write tcp 188.166.225.170:42026->77.90.31.13:4628: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42026->77.90.31.13:4628: use of closed network connection Nov 5 18:52:35.899860439 (151h12m7.595709192s) [noise=true , address=45.91.116.7:4010 , piece_id=EBVGGNQF3N4MTUIATJUWK4QZN7SEOWI6SIAW6N7JZOB5G6H33NEQ, seg_pos={0 0}, upload=133242]: Done uploading piece. err: ecclient: upload failed (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, address:45.91.116.7:4010): protocol: write tcp 188.166.225.170:40036->45.91.116.7:4010: use of closed network connection; write tcp 188.166.225.170:40036->45.91.116.7:4010: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40036->45.91.116.7:4010: use of closed network connection Nov 5 18:52:35.899923829 (151h12m7.595772572s) [noise=true , address=77.90.31.18:4805 , piece_id=U2U34KK4AJK4A5HGNTHEYCIARLAR4X5NLOVCF63TN3PJUIRMMIHA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: write tcp 188.166.225.170:60758->77.90.31.18:4805: use of closed network connection; write tcp 188.166.225.170:60758->77.90.31.18:4805: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60758->77.90.31.18:4805: use of closed network connection Nov 5 18:52:35.900052084 (151h12m7.595900837s) [noise=true , address=77.90.31.16:4734 , piece_id=HFWVFM46PZFLZT6YNZVIA6HKLVN2PTV2GETZEKX356DMDGQO4HOA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: write tcp 188.166.225.170:50728->77.90.31.16:4734: use of closed network connection; write tcp 188.166.225.170:50728->77.90.31.16:4734: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50728->77.90.31.16:4734: use of closed network connection Nov 5 18:52:35.900231175 (151h12m7.596079938s) [noise=true , address=45.91.116.11:4117 , piece_id=QUGUYHEJWEEV733CBC3ONMV7XWDKVH2OFLQWXPQHPKZ7GTH4RLUA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: ecclient: upload failed (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, address:45.91.116.11:4117): protocol: write tcp 188.166.225.170:41292->45.91.116.11:4117: use of closed network connection; write tcp 188.166.225.170:41292->45.91.116.11:4117: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41292->45.91.116.11:4117: use of closed network connection Nov 5 18:52:35.900342116 (151h12m7.596190859s) [noise=true , address=77.90.31.15:4698 , piece_id=7VACL26N7ZCBXIA2I3XL5Q3UJ2BAHDIPXX4VZCQX3S63ABNW625Q, seg_pos={0 0}, upload=133242]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: write tcp 188.166.225.170:41118->77.90.31.15:4698: use of closed network connection; write tcp 188.166.225.170:41118->77.90.31.15:4698: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41118->77.90.31.15:4698: use of closed network connection Nov 5 18:52:35.900382814 (151h12m7.596231567s) [seg_pos={0 0}, upload=133242]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Nov 5 18:52:36.162442033 (151h12m7.858290786s) [upload=133242]: Done uploading segments... Nov 5 18:52:36.162481568 (151h12m7.858330321s) [upload=133242]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 6.005591597s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.................X...X......................................] duration: 6.167565194s trace: https://jaeger.dev.storj.io/trace/74cf1e30f6d49950 FAIL (2025-11-05 19:20:48 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.167595892s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [................X..X........................................] duration: 5.685469867s trace: https://jaeger.dev.storj.io/trace/3e2ae8884260b157 FAIL (2025-11-05 19:20:48 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.685564237s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...............X...X........................................] duration: 5.974943119s trace: https://jaeger.dev.storj.io/trace/2e576aae48e8f03e FAIL (2025-11-05 19:20:38 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== 5 18:52:06.850167887 (151h11m38.54601663s) [noise=true , address=145.79.227.25:4618 , piece_id=XRUPY5BUPG7BXN2MI6MGYO4N3GW6TZDJE46MZFNRELWANHHG3N5A, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.850747301 (151h11m38.546596054s) [noise=true , address=77.90.31.15:4698 , piece_id=7VACL26N7ZCBXIA2I3XL5Q3UJ2BAHDIPXX4VZCQX3S63ABNW625Q, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.851222605 (151h11m38.547071358s) [noise=true , address=145.79.227.17:4972 , piece_id=H7UHQV7RDLCZUHY2XMXUAD2PX4ZBLWKTM47N6PH7UOGBRXPMWVOA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.851558575 (151h11m38.547407319s) [noise=true , address=145.79.227.22:4509 , piece_id=7RA5NJ72PI3DT4IC2ECVLZ3A2HXXL546ELVDXHZW4OXNN4D36OEA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.851909263 (151h11m38.547758006s) [noise=true , address=145.79.227.26:4654 , piece_id=E5J22KNPQSETSY443QQZXFFKUUJFHWFV22SQAVZ4BFQ4M6YWCNQQ, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.852327540 (151h11m38.548176293s) [noise=true , address=45.91.116.7:4010 , piece_id=EBVGGNQF3N4MTUIATJUWK4QZN7SEOWI6SIAW6N7JZOB5G6H33NEQ, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.852728624 (151h11m38.548577377s) [noise=true , address=145.79.227.9:4366 , piece_id=7ZRECKBQC4S6QLERBZY442WYS4H35XWOZYSGGPHSU7NQ6WL4UNVA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.853978774 (151h11m38.549827528s) [noise=true , address=77.90.31.18:4805 , piece_id=U2U34KK4AJK4A5HGNTHEYCIARLAR4X5NLOVCF63TN3PJUIRMMIHA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.854447296 (151h11m38.550296059s) [noise=true , address=77.90.31.16:4734 , piece_id=HFWVFM46PZFLZT6YNZVIA6HKLVN2PTV2GETZEKX356DMDGQO4HOA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.856523089 (151h11m38.552371842s) [noise=true , address=45.91.116.11:4117 , piece_id=QUGUYHEJWEEV733CBC3ONMV7XWDKVH2OFLQWXPQHPKZ7GTH4RLUA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.859490379 (151h11m38.555339122s) [noise=true , address=77.90.31.13:4628 , piece_id=QFTEHZ5FY7A5DPZZ44TRXGFRHPRNJZJHLQZDGRGOWYFEZYIZT2NQ, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.861350291 (151h11m38.557199034s) [noise=true , address=145.79.227.31:4797 , piece_id=2TFSZNURIAU573QUA5DUS7AJZNY2OXC6TJL3ST4VXIWY2WVB5YJA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.863368665 (151h11m38.559217418s) [noise=true , address=145.79.227.23:4547 , piece_id=K54H437KW5AAYUWOTMDBDP5GJ3ZABFKLAAOSS6WS5G5CROYEHODQ, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.864848794 (151h11m38.560697547s) [noise=true , address=145.79.227.24:4581 , piece_id=TSCVCH2KJEVMD4AVOEIHGTMRCRZ6NZMZA2CTQ6FUO2EIL3JAVYEA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.865644930 (151h11m38.561493683s) [noise=true , address=145.79.227.8:4330 , piece_id=4A47HIUX2MTJ2673XZZCMMF4T235TS4JWMNYJEVHHCKAHQX6APMA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:08.560504137 (151h11m40.25635289s) [upload=133242]: Next returned nil segment Nov 5 18:52:08.560522693 (151h11m40.256371446s) [upload=133242]: Waiting for error group managing segments... Nov 5 18:52:08.850333179 (151h11m40.546181932s) [noise=true , address=145.79.227.17:4972 , piece_id=H7UHQV7RDLCZUHY2XMXUAD2PX4ZBLWKTM47N6PH7UOGBRXPMWVOA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.853517822 (151h11m40.549366575s) [noise=true , address=145.79.227.22:4509 , piece_id=7RA5NJ72PI3DT4IC2ECVLZ3A2HXXL546ELVDXHZW4OXNN4D36OEA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.855547508 (151h11m40.551396261s) [noise=true , address=145.79.227.9:4366 , piece_id=7ZRECKBQC4S6QLERBZY442WYS4H35XWOZYSGGPHSU7NQ6WL4UNVA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.856103106 (151h11m40.551951859s) [noise=true , address=145.79.227.25:4618 , piece_id=XRUPY5BUPG7BXN2MI6MGYO4N3GW6TZDJE46MZFNRELWANHHG3N5A, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.860417321 (151h11m40.556266064s) [noise=true , address=145.79.227.8:4330 , piece_id=4A47HIUX2MTJ2673XZZCMMF4T235TS4JWMNYJEVHHCKAHQX6APMA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.865602796 (151h11m40.561451549s) [noise=true , address=145.79.227.23:4547 , piece_id=K54H437KW5AAYUWOTMDBDP5GJ3ZABFKLAAOSS6WS5G5CROYEHODQ, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.879393757 (151h11m40.57524251s) [noise=true , address=145.79.227.31:4797 , piece_id=2TFSZNURIAU573QUA5DUS7AJZNY2OXC6TJL3ST4VXIWY2WVB5YJA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.905541449 (151h11m40.601390202s) [noise=true , address=145.79.227.26:4654 , piece_id=E5J22KNPQSETSY443QQZXFFKUUJFHWFV22SQAVZ4BFQ4M6YWCNQQ, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.929018836 (151h11m40.624867589s) [noise=true , address=145.79.227.24:4581 , piece_id=TSCVCH2KJEVMD4AVOEIHGTMRCRZ6NZMZA2CTQ6FUO2EIL3JAVYEA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:35.899554917 (151h12m7.59540368s) [noise=true , address=77.90.31.17:4771 , piece_id=KVFE7VYW3RZJ25BWSH4BIRRQCMWDYOIK7EJ62XVREGN26YN7XYMQ, seg_pos={0 0}, upload=133242]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: write tcp 188.166.225.170:60676->77.90.31.17:4771: use of closed network connection; write tcp 188.166.225.170:60676->77.90.31.17:4771: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60676->77.90.31.17:4771: use of closed network connection Nov 5 18:52:35.899727506 (151h12m7.595576259s) [noise=true , address=77.90.31.13:4628 , piece_id=QFTEHZ5FY7A5DPZZ44TRXGFRHPRNJZJHLQZDGRGOWYFEZYIZT2NQ, seg_pos={0 0}, upload=133242]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: write tcp 188.166.225.170:42026->77.90.31.13:4628: use of closed network connection; write tcp 188.166.225.170:42026->77.90.31.13:4628: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42026->77.90.31.13:4628: use of closed network connection Nov 5 18:52:35.899860439 (151h12m7.595709192s) [noise=true , address=45.91.116.7:4010 , piece_id=EBVGGNQF3N4MTUIATJUWK4QZN7SEOWI6SIAW6N7JZOB5G6H33NEQ, seg_pos={0 0}, upload=133242]: Done uploading piece. err: ecclient: upload failed (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, address:45.91.116.7:4010): protocol: write tcp 188.166.225.170:40036->45.91.116.7:4010: use of closed network connection; write tcp 188.166.225.170:40036->45.91.116.7:4010: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40036->45.91.116.7:4010: use of closed network connection Nov 5 18:52:35.899923829 (151h12m7.595772572s) [noise=true , address=77.90.31.18:4805 , piece_id=U2U34KK4AJK4A5HGNTHEYCIARLAR4X5NLOVCF63TN3PJUIRMMIHA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: write tcp 188.166.225.170:60758->77.90.31.18:4805: use of closed network connection; write tcp 188.166.225.170:60758->77.90.31.18:4805: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60758->77.90.31.18:4805: use of closed network connection Nov 5 18:52:35.900052084 (151h12m7.595900837s) [noise=true , address=77.90.31.16:4734 , piece_id=HFWVFM46PZFLZT6YNZVIA6HKLVN2PTV2GETZEKX356DMDGQO4HOA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: write tcp 188.166.225.170:50728->77.90.31.16:4734: use of closed network connection; write tcp 188.166.225.170:50728->77.90.31.16:4734: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50728->77.90.31.16:4734: use of closed network connection Nov 5 18:52:35.900231175 (151h12m7.596079938s) [noise=true , address=45.91.116.11:4117 , piece_id=QUGUYHEJWEEV733CBC3ONMV7XWDKVH2OFLQWXPQHPKZ7GTH4RLUA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: ecclient: upload failed (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, address:45.91.116.11:4117): protocol: write tcp 188.166.225.170:41292->45.91.116.11:4117: use of closed network connection; write tcp 188.166.225.170:41292->45.91.116.11:4117: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41292->45.91.116.11:4117: use of closed network connection Nov 5 18:52:35.900342116 (151h12m7.596190859s) [noise=true , address=77.90.31.15:4698 , piece_id=7VACL26N7ZCBXIA2I3XL5Q3UJ2BAHDIPXX4VZCQX3S63ABNW625Q, seg_pos={0 0}, upload=133242]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: write tcp 188.166.225.170:41118->77.90.31.15:4698: use of closed network connection; write tcp 188.166.225.170:41118->77.90.31.15:4698: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41118->77.90.31.15:4698: use of closed network connection Nov 5 18:52:35.900382814 (151h12m7.596231567s) [seg_pos={0 0}, upload=133242]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Nov 5 18:52:36.162442033 (151h12m7.858290786s) [upload=133242]: Done uploading segments... Nov 5 18:52:36.162481568 (151h12m7.858330321s) [upload=133242]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 5.974972095s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [..................XX...X....................................] duration: 3.455332304s trace: https://jaeger.dev.storj.io/trace/204ab2570dc82d9c FAIL (2025-11-05 19:20:16 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.455423196s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [................X...X.......................................] duration: 6.223903781s trace: https://jaeger.dev.storj.io/trace/2e576aae48e8f03e FAIL (2025-11-05 19:20:08 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== 5 18:52:06.850167887 (151h11m38.54601663s) [noise=true , address=145.79.227.25:4618 , piece_id=XRUPY5BUPG7BXN2MI6MGYO4N3GW6TZDJE46MZFNRELWANHHG3N5A, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.850747301 (151h11m38.546596054s) [noise=true , address=77.90.31.15:4698 , piece_id=7VACL26N7ZCBXIA2I3XL5Q3UJ2BAHDIPXX4VZCQX3S63ABNW625Q, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.851222605 (151h11m38.547071358s) [noise=true , address=145.79.227.17:4972 , piece_id=H7UHQV7RDLCZUHY2XMXUAD2PX4ZBLWKTM47N6PH7UOGBRXPMWVOA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.851558575 (151h11m38.547407319s) [noise=true , address=145.79.227.22:4509 , piece_id=7RA5NJ72PI3DT4IC2ECVLZ3A2HXXL546ELVDXHZW4OXNN4D36OEA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.851909263 (151h11m38.547758006s) [noise=true , address=145.79.227.26:4654 , piece_id=E5J22KNPQSETSY443QQZXFFKUUJFHWFV22SQAVZ4BFQ4M6YWCNQQ, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.852327540 (151h11m38.548176293s) [noise=true , address=45.91.116.7:4010 , piece_id=EBVGGNQF3N4MTUIATJUWK4QZN7SEOWI6SIAW6N7JZOB5G6H33NEQ, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.852728624 (151h11m38.548577377s) [noise=true , address=145.79.227.9:4366 , piece_id=7ZRECKBQC4S6QLERBZY442WYS4H35XWOZYSGGPHSU7NQ6WL4UNVA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.853978774 (151h11m38.549827528s) [noise=true , address=77.90.31.18:4805 , piece_id=U2U34KK4AJK4A5HGNTHEYCIARLAR4X5NLOVCF63TN3PJUIRMMIHA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.854447296 (151h11m38.550296059s) [noise=true , address=77.90.31.16:4734 , piece_id=HFWVFM46PZFLZT6YNZVIA6HKLVN2PTV2GETZEKX356DMDGQO4HOA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.856523089 (151h11m38.552371842s) [noise=true , address=45.91.116.11:4117 , piece_id=QUGUYHEJWEEV733CBC3ONMV7XWDKVH2OFLQWXPQHPKZ7GTH4RLUA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.859490379 (151h11m38.555339122s) [noise=true , address=77.90.31.13:4628 , piece_id=QFTEHZ5FY7A5DPZZ44TRXGFRHPRNJZJHLQZDGRGOWYFEZYIZT2NQ, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.861350291 (151h11m38.557199034s) [noise=true , address=145.79.227.31:4797 , piece_id=2TFSZNURIAU573QUA5DUS7AJZNY2OXC6TJL3ST4VXIWY2WVB5YJA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.863368665 (151h11m38.559217418s) [noise=true , address=145.79.227.23:4547 , piece_id=K54H437KW5AAYUWOTMDBDP5GJ3ZABFKLAAOSS6WS5G5CROYEHODQ, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.864848794 (151h11m38.560697547s) [noise=true , address=145.79.227.24:4581 , piece_id=TSCVCH2KJEVMD4AVOEIHGTMRCRZ6NZMZA2CTQ6FUO2EIL3JAVYEA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.865644930 (151h11m38.561493683s) [noise=true , address=145.79.227.8:4330 , piece_id=4A47HIUX2MTJ2673XZZCMMF4T235TS4JWMNYJEVHHCKAHQX6APMA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:08.560504137 (151h11m40.25635289s) [upload=133242]: Next returned nil segment Nov 5 18:52:08.560522693 (151h11m40.256371446s) [upload=133242]: Waiting for error group managing segments... Nov 5 18:52:08.850333179 (151h11m40.546181932s) [noise=true , address=145.79.227.17:4972 , piece_id=H7UHQV7RDLCZUHY2XMXUAD2PX4ZBLWKTM47N6PH7UOGBRXPMWVOA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.853517822 (151h11m40.549366575s) [noise=true , address=145.79.227.22:4509 , piece_id=7RA5NJ72PI3DT4IC2ECVLZ3A2HXXL546ELVDXHZW4OXNN4D36OEA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.855547508 (151h11m40.551396261s) [noise=true , address=145.79.227.9:4366 , piece_id=7ZRECKBQC4S6QLERBZY442WYS4H35XWOZYSGGPHSU7NQ6WL4UNVA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.856103106 (151h11m40.551951859s) [noise=true , address=145.79.227.25:4618 , piece_id=XRUPY5BUPG7BXN2MI6MGYO4N3GW6TZDJE46MZFNRELWANHHG3N5A, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.860417321 (151h11m40.556266064s) [noise=true , address=145.79.227.8:4330 , piece_id=4A47HIUX2MTJ2673XZZCMMF4T235TS4JWMNYJEVHHCKAHQX6APMA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.865602796 (151h11m40.561451549s) [noise=true , address=145.79.227.23:4547 , piece_id=K54H437KW5AAYUWOTMDBDP5GJ3ZABFKLAAOSS6WS5G5CROYEHODQ, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.879393757 (151h11m40.57524251s) [noise=true , address=145.79.227.31:4797 , piece_id=2TFSZNURIAU573QUA5DUS7AJZNY2OXC6TJL3ST4VXIWY2WVB5YJA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.905541449 (151h11m40.601390202s) [noise=true , address=145.79.227.26:4654 , piece_id=E5J22KNPQSETSY443QQZXFFKUUJFHWFV22SQAVZ4BFQ4M6YWCNQQ, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.929018836 (151h11m40.624867589s) [noise=true , address=145.79.227.24:4581 , piece_id=TSCVCH2KJEVMD4AVOEIHGTMRCRZ6NZMZA2CTQ6FUO2EIL3JAVYEA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:35.899554917 (151h12m7.59540368s) [noise=true , address=77.90.31.17:4771 , piece_id=KVFE7VYW3RZJ25BWSH4BIRRQCMWDYOIK7EJ62XVREGN26YN7XYMQ, seg_pos={0 0}, upload=133242]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: write tcp 188.166.225.170:60676->77.90.31.17:4771: use of closed network connection; write tcp 188.166.225.170:60676->77.90.31.17:4771: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60676->77.90.31.17:4771: use of closed network connection Nov 5 18:52:35.899727506 (151h12m7.595576259s) [noise=true , address=77.90.31.13:4628 , piece_id=QFTEHZ5FY7A5DPZZ44TRXGFRHPRNJZJHLQZDGRGOWYFEZYIZT2NQ, seg_pos={0 0}, upload=133242]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: write tcp 188.166.225.170:42026->77.90.31.13:4628: use of closed network connection; write tcp 188.166.225.170:42026->77.90.31.13:4628: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42026->77.90.31.13:4628: use of closed network connection Nov 5 18:52:35.899860439 (151h12m7.595709192s) [noise=true , address=45.91.116.7:4010 , piece_id=EBVGGNQF3N4MTUIATJUWK4QZN7SEOWI6SIAW6N7JZOB5G6H33NEQ, seg_pos={0 0}, upload=133242]: Done uploading piece. err: ecclient: upload failed (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, address:45.91.116.7:4010): protocol: write tcp 188.166.225.170:40036->45.91.116.7:4010: use of closed network connection; write tcp 188.166.225.170:40036->45.91.116.7:4010: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40036->45.91.116.7:4010: use of closed network connection Nov 5 18:52:35.899923829 (151h12m7.595772572s) [noise=true , address=77.90.31.18:4805 , piece_id=U2U34KK4AJK4A5HGNTHEYCIARLAR4X5NLOVCF63TN3PJUIRMMIHA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: write tcp 188.166.225.170:60758->77.90.31.18:4805: use of closed network connection; write tcp 188.166.225.170:60758->77.90.31.18:4805: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60758->77.90.31.18:4805: use of closed network connection Nov 5 18:52:35.900052084 (151h12m7.595900837s) [noise=true , address=77.90.31.16:4734 , piece_id=HFWVFM46PZFLZT6YNZVIA6HKLVN2PTV2GETZEKX356DMDGQO4HOA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: write tcp 188.166.225.170:50728->77.90.31.16:4734: use of closed network connection; write tcp 188.166.225.170:50728->77.90.31.16:4734: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50728->77.90.31.16:4734: use of closed network connection Nov 5 18:52:35.900231175 (151h12m7.596079938s) [noise=true , address=45.91.116.11:4117 , piece_id=QUGUYHEJWEEV733CBC3ONMV7XWDKVH2OFLQWXPQHPKZ7GTH4RLUA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: ecclient: upload failed (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, address:45.91.116.11:4117): protocol: write tcp 188.166.225.170:41292->45.91.116.11:4117: use of closed network connection; write tcp 188.166.225.170:41292->45.91.116.11:4117: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41292->45.91.116.11:4117: use of closed network connection Nov 5 18:52:35.900342116 (151h12m7.596190859s) [noise=true , address=77.90.31.15:4698 , piece_id=7VACL26N7ZCBXIA2I3XL5Q3UJ2BAHDIPXX4VZCQX3S63ABNW625Q, seg_pos={0 0}, upload=133242]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: write tcp 188.166.225.170:41118->77.90.31.15:4698: use of closed network connection; write tcp 188.166.225.170:41118->77.90.31.15:4698: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41118->77.90.31.15:4698: use of closed network connection Nov 5 18:52:35.900382814 (151h12m7.596231567s) [seg_pos={0 0}, upload=133242]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Nov 5 18:52:36.162442033 (151h12m7.858290786s) [upload=133242]: Done uploading segments... Nov 5 18:52:36.162481568 (151h12m7.858330321s) [upload=133242]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 6.224146392s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..................X...X.....................................] duration: 6.482270482s trace: https://jaeger.dev.storj.io/trace/74cf1e30f6d49950 FAIL (2025-11-05 19:20:03 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.482298696s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [...................XX...X...................................] duration: 3.484475389s trace: https://jaeger.dev.storj.io/trace/204ab2570dc82d9c FAIL (2025-11-05 19:20:01 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.484508751s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.................X...X......................................] duration: 5.92393281s trace: https://jaeger.dev.storj.io/trace/2e576aae48e8f03e FAIL (2025-11-05 19:19:52 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== 5 18:52:06.850167887 (151h11m38.54601663s) [noise=true , address=145.79.227.25:4618 , piece_id=XRUPY5BUPG7BXN2MI6MGYO4N3GW6TZDJE46MZFNRELWANHHG3N5A, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.850747301 (151h11m38.546596054s) [noise=true , address=77.90.31.15:4698 , piece_id=7VACL26N7ZCBXIA2I3XL5Q3UJ2BAHDIPXX4VZCQX3S63ABNW625Q, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.851222605 (151h11m38.547071358s) [noise=true , address=145.79.227.17:4972 , piece_id=H7UHQV7RDLCZUHY2XMXUAD2PX4ZBLWKTM47N6PH7UOGBRXPMWVOA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.851558575 (151h11m38.547407319s) [noise=true , address=145.79.227.22:4509 , piece_id=7RA5NJ72PI3DT4IC2ECVLZ3A2HXXL546ELVDXHZW4OXNN4D36OEA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.851909263 (151h11m38.547758006s) [noise=true , address=145.79.227.26:4654 , piece_id=E5J22KNPQSETSY443QQZXFFKUUJFHWFV22SQAVZ4BFQ4M6YWCNQQ, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.852327540 (151h11m38.548176293s) [noise=true , address=45.91.116.7:4010 , piece_id=EBVGGNQF3N4MTUIATJUWK4QZN7SEOWI6SIAW6N7JZOB5G6H33NEQ, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.852728624 (151h11m38.548577377s) [noise=true , address=145.79.227.9:4366 , piece_id=7ZRECKBQC4S6QLERBZY442WYS4H35XWOZYSGGPHSU7NQ6WL4UNVA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.853978774 (151h11m38.549827528s) [noise=true , address=77.90.31.18:4805 , piece_id=U2U34KK4AJK4A5HGNTHEYCIARLAR4X5NLOVCF63TN3PJUIRMMIHA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.854447296 (151h11m38.550296059s) [noise=true , address=77.90.31.16:4734 , piece_id=HFWVFM46PZFLZT6YNZVIA6HKLVN2PTV2GETZEKX356DMDGQO4HOA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.856523089 (151h11m38.552371842s) [noise=true , address=45.91.116.11:4117 , piece_id=QUGUYHEJWEEV733CBC3ONMV7XWDKVH2OFLQWXPQHPKZ7GTH4RLUA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.859490379 (151h11m38.555339122s) [noise=true , address=77.90.31.13:4628 , piece_id=QFTEHZ5FY7A5DPZZ44TRXGFRHPRNJZJHLQZDGRGOWYFEZYIZT2NQ, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.861350291 (151h11m38.557199034s) [noise=true , address=145.79.227.31:4797 , piece_id=2TFSZNURIAU573QUA5DUS7AJZNY2OXC6TJL3ST4VXIWY2WVB5YJA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.863368665 (151h11m38.559217418s) [noise=true , address=145.79.227.23:4547 , piece_id=K54H437KW5AAYUWOTMDBDP5GJ3ZABFKLAAOSS6WS5G5CROYEHODQ, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.864848794 (151h11m38.560697547s) [noise=true , address=145.79.227.24:4581 , piece_id=TSCVCH2KJEVMD4AVOEIHGTMRCRZ6NZMZA2CTQ6FUO2EIL3JAVYEA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.865644930 (151h11m38.561493683s) [noise=true , address=145.79.227.8:4330 , piece_id=4A47HIUX2MTJ2673XZZCMMF4T235TS4JWMNYJEVHHCKAHQX6APMA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:08.560504137 (151h11m40.25635289s) [upload=133242]: Next returned nil segment Nov 5 18:52:08.560522693 (151h11m40.256371446s) [upload=133242]: Waiting for error group managing segments... Nov 5 18:52:08.850333179 (151h11m40.546181932s) [noise=true , address=145.79.227.17:4972 , piece_id=H7UHQV7RDLCZUHY2XMXUAD2PX4ZBLWKTM47N6PH7UOGBRXPMWVOA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.853517822 (151h11m40.549366575s) [noise=true , address=145.79.227.22:4509 , piece_id=7RA5NJ72PI3DT4IC2ECVLZ3A2HXXL546ELVDXHZW4OXNN4D36OEA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.855547508 (151h11m40.551396261s) [noise=true , address=145.79.227.9:4366 , piece_id=7ZRECKBQC4S6QLERBZY442WYS4H35XWOZYSGGPHSU7NQ6WL4UNVA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.856103106 (151h11m40.551951859s) [noise=true , address=145.79.227.25:4618 , piece_id=XRUPY5BUPG7BXN2MI6MGYO4N3GW6TZDJE46MZFNRELWANHHG3N5A, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.860417321 (151h11m40.556266064s) [noise=true , address=145.79.227.8:4330 , piece_id=4A47HIUX2MTJ2673XZZCMMF4T235TS4JWMNYJEVHHCKAHQX6APMA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.865602796 (151h11m40.561451549s) [noise=true , address=145.79.227.23:4547 , piece_id=K54H437KW5AAYUWOTMDBDP5GJ3ZABFKLAAOSS6WS5G5CROYEHODQ, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.879393757 (151h11m40.57524251s) [noise=true , address=145.79.227.31:4797 , piece_id=2TFSZNURIAU573QUA5DUS7AJZNY2OXC6TJL3ST4VXIWY2WVB5YJA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.905541449 (151h11m40.601390202s) [noise=true , address=145.79.227.26:4654 , piece_id=E5J22KNPQSETSY443QQZXFFKUUJFHWFV22SQAVZ4BFQ4M6YWCNQQ, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.929018836 (151h11m40.624867589s) [noise=true , address=145.79.227.24:4581 , piece_id=TSCVCH2KJEVMD4AVOEIHGTMRCRZ6NZMZA2CTQ6FUO2EIL3JAVYEA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:35.899554917 (151h12m7.59540368s) [noise=true , address=77.90.31.17:4771 , piece_id=KVFE7VYW3RZJ25BWSH4BIRRQCMWDYOIK7EJ62XVREGN26YN7XYMQ, seg_pos={0 0}, upload=133242]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: write tcp 188.166.225.170:60676->77.90.31.17:4771: use of closed network connection; write tcp 188.166.225.170:60676->77.90.31.17:4771: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60676->77.90.31.17:4771: use of closed network connection Nov 5 18:52:35.899727506 (151h12m7.595576259s) [noise=true , address=77.90.31.13:4628 , piece_id=QFTEHZ5FY7A5DPZZ44TRXGFRHPRNJZJHLQZDGRGOWYFEZYIZT2NQ, seg_pos={0 0}, upload=133242]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: write tcp 188.166.225.170:42026->77.90.31.13:4628: use of closed network connection; write tcp 188.166.225.170:42026->77.90.31.13:4628: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42026->77.90.31.13:4628: use of closed network connection Nov 5 18:52:35.899860439 (151h12m7.595709192s) [noise=true , address=45.91.116.7:4010 , piece_id=EBVGGNQF3N4MTUIATJUWK4QZN7SEOWI6SIAW6N7JZOB5G6H33NEQ, seg_pos={0 0}, upload=133242]: Done uploading piece. err: ecclient: upload failed (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, address:45.91.116.7:4010): protocol: write tcp 188.166.225.170:40036->45.91.116.7:4010: use of closed network connection; write tcp 188.166.225.170:40036->45.91.116.7:4010: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40036->45.91.116.7:4010: use of closed network connection Nov 5 18:52:35.899923829 (151h12m7.595772572s) [noise=true , address=77.90.31.18:4805 , piece_id=U2U34KK4AJK4A5HGNTHEYCIARLAR4X5NLOVCF63TN3PJUIRMMIHA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: write tcp 188.166.225.170:60758->77.90.31.18:4805: use of closed network connection; write tcp 188.166.225.170:60758->77.90.31.18:4805: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60758->77.90.31.18:4805: use of closed network connection Nov 5 18:52:35.900052084 (151h12m7.595900837s) [noise=true , address=77.90.31.16:4734 , piece_id=HFWVFM46PZFLZT6YNZVIA6HKLVN2PTV2GETZEKX356DMDGQO4HOA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: write tcp 188.166.225.170:50728->77.90.31.16:4734: use of closed network connection; write tcp 188.166.225.170:50728->77.90.31.16:4734: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50728->77.90.31.16:4734: use of closed network connection Nov 5 18:52:35.900231175 (151h12m7.596079938s) [noise=true , address=45.91.116.11:4117 , piece_id=QUGUYHEJWEEV733CBC3ONMV7XWDKVH2OFLQWXPQHPKZ7GTH4RLUA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: ecclient: upload failed (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, address:45.91.116.11:4117): protocol: write tcp 188.166.225.170:41292->45.91.116.11:4117: use of closed network connection; write tcp 188.166.225.170:41292->45.91.116.11:4117: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41292->45.91.116.11:4117: use of closed network connection Nov 5 18:52:35.900342116 (151h12m7.596190859s) [noise=true , address=77.90.31.15:4698 , piece_id=7VACL26N7ZCBXIA2I3XL5Q3UJ2BAHDIPXX4VZCQX3S63ABNW625Q, seg_pos={0 0}, upload=133242]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: write tcp 188.166.225.170:41118->77.90.31.15:4698: use of closed network connection; write tcp 188.166.225.170:41118->77.90.31.15:4698: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41118->77.90.31.15:4698: use of closed network connection Nov 5 18:52:35.900382814 (151h12m7.596231567s) [seg_pos={0 0}, upload=133242]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Nov 5 18:52:36.162442033 (151h12m7.858290786s) [upload=133242]: Done uploading segments... Nov 5 18:52:36.162481568 (151h12m7.858330321s) [upload=133242]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 5.924051156s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.................X..X.......................................] duration: 5.75919183s trace: https://jaeger.dev.storj.io/trace/3e2ae8884260b157 FAIL (2025-11-05 19:19:45 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.759229732s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...................X...X....................................] duration: 6.239986005s trace: https://jaeger.dev.storj.io/trace/74cf1e30f6d49950 FAIL (2025-11-05 19:19:35 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.240008659s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [..................X..X......................................] duration: 5.72691526s trace: https://jaeger.dev.storj.io/trace/3e2ae8884260b157 FAIL (2025-11-05 19:19:30 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.726948263s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..................X...X.....................................] duration: 5.935915433s trace: https://jaeger.dev.storj.io/trace/2e576aae48e8f03e FAIL (2025-11-05 19:19:12 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== 5 18:52:06.850167887 (151h11m38.54601663s) [noise=true , address=145.79.227.25:4618 , piece_id=XRUPY5BUPG7BXN2MI6MGYO4N3GW6TZDJE46MZFNRELWANHHG3N5A, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.850747301 (151h11m38.546596054s) [noise=true , address=77.90.31.15:4698 , piece_id=7VACL26N7ZCBXIA2I3XL5Q3UJ2BAHDIPXX4VZCQX3S63ABNW625Q, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.851222605 (151h11m38.547071358s) [noise=true , address=145.79.227.17:4972 , piece_id=H7UHQV7RDLCZUHY2XMXUAD2PX4ZBLWKTM47N6PH7UOGBRXPMWVOA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.851558575 (151h11m38.547407319s) [noise=true , address=145.79.227.22:4509 , piece_id=7RA5NJ72PI3DT4IC2ECVLZ3A2HXXL546ELVDXHZW4OXNN4D36OEA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.851909263 (151h11m38.547758006s) [noise=true , address=145.79.227.26:4654 , piece_id=E5J22KNPQSETSY443QQZXFFKUUJFHWFV22SQAVZ4BFQ4M6YWCNQQ, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.852327540 (151h11m38.548176293s) [noise=true , address=45.91.116.7:4010 , piece_id=EBVGGNQF3N4MTUIATJUWK4QZN7SEOWI6SIAW6N7JZOB5G6H33NEQ, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.852728624 (151h11m38.548577377s) [noise=true , address=145.79.227.9:4366 , piece_id=7ZRECKBQC4S6QLERBZY442WYS4H35XWOZYSGGPHSU7NQ6WL4UNVA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.853978774 (151h11m38.549827528s) [noise=true , address=77.90.31.18:4805 , piece_id=U2U34KK4AJK4A5HGNTHEYCIARLAR4X5NLOVCF63TN3PJUIRMMIHA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.854447296 (151h11m38.550296059s) [noise=true , address=77.90.31.16:4734 , piece_id=HFWVFM46PZFLZT6YNZVIA6HKLVN2PTV2GETZEKX356DMDGQO4HOA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.856523089 (151h11m38.552371842s) [noise=true , address=45.91.116.11:4117 , piece_id=QUGUYHEJWEEV733CBC3ONMV7XWDKVH2OFLQWXPQHPKZ7GTH4RLUA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.859490379 (151h11m38.555339122s) [noise=true , address=77.90.31.13:4628 , piece_id=QFTEHZ5FY7A5DPZZ44TRXGFRHPRNJZJHLQZDGRGOWYFEZYIZT2NQ, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.861350291 (151h11m38.557199034s) [noise=true , address=145.79.227.31:4797 , piece_id=2TFSZNURIAU573QUA5DUS7AJZNY2OXC6TJL3ST4VXIWY2WVB5YJA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.863368665 (151h11m38.559217418s) [noise=true , address=145.79.227.23:4547 , piece_id=K54H437KW5AAYUWOTMDBDP5GJ3ZABFKLAAOSS6WS5G5CROYEHODQ, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.864848794 (151h11m38.560697547s) [noise=true , address=145.79.227.24:4581 , piece_id=TSCVCH2KJEVMD4AVOEIHGTMRCRZ6NZMZA2CTQ6FUO2EIL3JAVYEA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.865644930 (151h11m38.561493683s) [noise=true , address=145.79.227.8:4330 , piece_id=4A47HIUX2MTJ2673XZZCMMF4T235TS4JWMNYJEVHHCKAHQX6APMA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:08.560504137 (151h11m40.25635289s) [upload=133242]: Next returned nil segment Nov 5 18:52:08.560522693 (151h11m40.256371446s) [upload=133242]: Waiting for error group managing segments... Nov 5 18:52:08.850333179 (151h11m40.546181932s) [noise=true , address=145.79.227.17:4972 , piece_id=H7UHQV7RDLCZUHY2XMXUAD2PX4ZBLWKTM47N6PH7UOGBRXPMWVOA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.853517822 (151h11m40.549366575s) [noise=true , address=145.79.227.22:4509 , piece_id=7RA5NJ72PI3DT4IC2ECVLZ3A2HXXL546ELVDXHZW4OXNN4D36OEA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.855547508 (151h11m40.551396261s) [noise=true , address=145.79.227.9:4366 , piece_id=7ZRECKBQC4S6QLERBZY442WYS4H35XWOZYSGGPHSU7NQ6WL4UNVA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.856103106 (151h11m40.551951859s) [noise=true , address=145.79.227.25:4618 , piece_id=XRUPY5BUPG7BXN2MI6MGYO4N3GW6TZDJE46MZFNRELWANHHG3N5A, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.860417321 (151h11m40.556266064s) [noise=true , address=145.79.227.8:4330 , piece_id=4A47HIUX2MTJ2673XZZCMMF4T235TS4JWMNYJEVHHCKAHQX6APMA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.865602796 (151h11m40.561451549s) [noise=true , address=145.79.227.23:4547 , piece_id=K54H437KW5AAYUWOTMDBDP5GJ3ZABFKLAAOSS6WS5G5CROYEHODQ, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.879393757 (151h11m40.57524251s) [noise=true , address=145.79.227.31:4797 , piece_id=2TFSZNURIAU573QUA5DUS7AJZNY2OXC6TJL3ST4VXIWY2WVB5YJA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.905541449 (151h11m40.601390202s) [noise=true , address=145.79.227.26:4654 , piece_id=E5J22KNPQSETSY443QQZXFFKUUJFHWFV22SQAVZ4BFQ4M6YWCNQQ, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.929018836 (151h11m40.624867589s) [noise=true , address=145.79.227.24:4581 , piece_id=TSCVCH2KJEVMD4AVOEIHGTMRCRZ6NZMZA2CTQ6FUO2EIL3JAVYEA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:35.899554917 (151h12m7.59540368s) [noise=true , address=77.90.31.17:4771 , piece_id=KVFE7VYW3RZJ25BWSH4BIRRQCMWDYOIK7EJ62XVREGN26YN7XYMQ, seg_pos={0 0}, upload=133242]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: write tcp 188.166.225.170:60676->77.90.31.17:4771: use of closed network connection; write tcp 188.166.225.170:60676->77.90.31.17:4771: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60676->77.90.31.17:4771: use of closed network connection Nov 5 18:52:35.899727506 (151h12m7.595576259s) [noise=true , address=77.90.31.13:4628 , piece_id=QFTEHZ5FY7A5DPZZ44TRXGFRHPRNJZJHLQZDGRGOWYFEZYIZT2NQ, seg_pos={0 0}, upload=133242]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: write tcp 188.166.225.170:42026->77.90.31.13:4628: use of closed network connection; write tcp 188.166.225.170:42026->77.90.31.13:4628: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42026->77.90.31.13:4628: use of closed network connection Nov 5 18:52:35.899860439 (151h12m7.595709192s) [noise=true , address=45.91.116.7:4010 , piece_id=EBVGGNQF3N4MTUIATJUWK4QZN7SEOWI6SIAW6N7JZOB5G6H33NEQ, seg_pos={0 0}, upload=133242]: Done uploading piece. err: ecclient: upload failed (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, address:45.91.116.7:4010): protocol: write tcp 188.166.225.170:40036->45.91.116.7:4010: use of closed network connection; write tcp 188.166.225.170:40036->45.91.116.7:4010: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40036->45.91.116.7:4010: use of closed network connection Nov 5 18:52:35.899923829 (151h12m7.595772572s) [noise=true , address=77.90.31.18:4805 , piece_id=U2U34KK4AJK4A5HGNTHEYCIARLAR4X5NLOVCF63TN3PJUIRMMIHA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: write tcp 188.166.225.170:60758->77.90.31.18:4805: use of closed network connection; write tcp 188.166.225.170:60758->77.90.31.18:4805: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60758->77.90.31.18:4805: use of closed network connection Nov 5 18:52:35.900052084 (151h12m7.595900837s) [noise=true , address=77.90.31.16:4734 , piece_id=HFWVFM46PZFLZT6YNZVIA6HKLVN2PTV2GETZEKX356DMDGQO4HOA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: write tcp 188.166.225.170:50728->77.90.31.16:4734: use of closed network connection; write tcp 188.166.225.170:50728->77.90.31.16:4734: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50728->77.90.31.16:4734: use of closed network connection Nov 5 18:52:35.900231175 (151h12m7.596079938s) [noise=true , address=45.91.116.11:4117 , piece_id=QUGUYHEJWEEV733CBC3ONMV7XWDKVH2OFLQWXPQHPKZ7GTH4RLUA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: ecclient: upload failed (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, address:45.91.116.11:4117): protocol: write tcp 188.166.225.170:41292->45.91.116.11:4117: use of closed network connection; write tcp 188.166.225.170:41292->45.91.116.11:4117: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41292->45.91.116.11:4117: use of closed network connection Nov 5 18:52:35.900342116 (151h12m7.596190859s) [noise=true , address=77.90.31.15:4698 , piece_id=7VACL26N7ZCBXIA2I3XL5Q3UJ2BAHDIPXX4VZCQX3S63ABNW625Q, seg_pos={0 0}, upload=133242]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: write tcp 188.166.225.170:41118->77.90.31.15:4698: use of closed network connection; write tcp 188.166.225.170:41118->77.90.31.15:4698: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41118->77.90.31.15:4698: use of closed network connection Nov 5 18:52:35.900382814 (151h12m7.596231567s) [seg_pos={0 0}, upload=133242]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Nov 5 18:52:36.162442033 (151h12m7.858290786s) [upload=133242]: Done uploading segments... Nov 5 18:52:36.162481568 (151h12m7.858330321s) [upload=133242]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 5.935972824s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [....................XX...X..................................] duration: 3.45436612s trace: https://jaeger.dev.storj.io/trace/204ab2570dc82d9c FAIL (2025-11-05 19:19:09 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.454390456s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [...................X..X.....................................] duration: 5.720337232s trace: https://jaeger.dev.storj.io/trace/3e2ae8884260b157 FAIL (2025-11-05 19:19:04 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.720359374s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [....................X...X...................................] duration: 6.185044721s trace: https://jaeger.dev.storj.io/trace/74cf1e30f6d49950 FAIL (2025-11-05 19:18:49 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.185078617s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...................X...X....................................] duration: 5.953101284s trace: https://jaeger.dev.storj.io/trace/2e576aae48e8f03e FAIL (2025-11-05 19:18:42 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== 5 18:52:06.850167887 (151h11m38.54601663s) [noise=true , address=145.79.227.25:4618 , piece_id=XRUPY5BUPG7BXN2MI6MGYO4N3GW6TZDJE46MZFNRELWANHHG3N5A, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.850747301 (151h11m38.546596054s) [noise=true , address=77.90.31.15:4698 , piece_id=7VACL26N7ZCBXIA2I3XL5Q3UJ2BAHDIPXX4VZCQX3S63ABNW625Q, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.851222605 (151h11m38.547071358s) [noise=true , address=145.79.227.17:4972 , piece_id=H7UHQV7RDLCZUHY2XMXUAD2PX4ZBLWKTM47N6PH7UOGBRXPMWVOA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.851558575 (151h11m38.547407319s) [noise=true , address=145.79.227.22:4509 , piece_id=7RA5NJ72PI3DT4IC2ECVLZ3A2HXXL546ELVDXHZW4OXNN4D36OEA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.851909263 (151h11m38.547758006s) [noise=true , address=145.79.227.26:4654 , piece_id=E5J22KNPQSETSY443QQZXFFKUUJFHWFV22SQAVZ4BFQ4M6YWCNQQ, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.852327540 (151h11m38.548176293s) [noise=true , address=45.91.116.7:4010 , piece_id=EBVGGNQF3N4MTUIATJUWK4QZN7SEOWI6SIAW6N7JZOB5G6H33NEQ, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.852728624 (151h11m38.548577377s) [noise=true , address=145.79.227.9:4366 , piece_id=7ZRECKBQC4S6QLERBZY442WYS4H35XWOZYSGGPHSU7NQ6WL4UNVA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.853978774 (151h11m38.549827528s) [noise=true , address=77.90.31.18:4805 , piece_id=U2U34KK4AJK4A5HGNTHEYCIARLAR4X5NLOVCF63TN3PJUIRMMIHA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.854447296 (151h11m38.550296059s) [noise=true , address=77.90.31.16:4734 , piece_id=HFWVFM46PZFLZT6YNZVIA6HKLVN2PTV2GETZEKX356DMDGQO4HOA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.856523089 (151h11m38.552371842s) [noise=true , address=45.91.116.11:4117 , piece_id=QUGUYHEJWEEV733CBC3ONMV7XWDKVH2OFLQWXPQHPKZ7GTH4RLUA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.859490379 (151h11m38.555339122s) [noise=true , address=77.90.31.13:4628 , piece_id=QFTEHZ5FY7A5DPZZ44TRXGFRHPRNJZJHLQZDGRGOWYFEZYIZT2NQ, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.861350291 (151h11m38.557199034s) [noise=true , address=145.79.227.31:4797 , piece_id=2TFSZNURIAU573QUA5DUS7AJZNY2OXC6TJL3ST4VXIWY2WVB5YJA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.863368665 (151h11m38.559217418s) [noise=true , address=145.79.227.23:4547 , piece_id=K54H437KW5AAYUWOTMDBDP5GJ3ZABFKLAAOSS6WS5G5CROYEHODQ, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.864848794 (151h11m38.560697547s) [noise=true , address=145.79.227.24:4581 , piece_id=TSCVCH2KJEVMD4AVOEIHGTMRCRZ6NZMZA2CTQ6FUO2EIL3JAVYEA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.865644930 (151h11m38.561493683s) [noise=true , address=145.79.227.8:4330 , piece_id=4A47HIUX2MTJ2673XZZCMMF4T235TS4JWMNYJEVHHCKAHQX6APMA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:08.560504137 (151h11m40.25635289s) [upload=133242]: Next returned nil segment Nov 5 18:52:08.560522693 (151h11m40.256371446s) [upload=133242]: Waiting for error group managing segments... Nov 5 18:52:08.850333179 (151h11m40.546181932s) [noise=true , address=145.79.227.17:4972 , piece_id=H7UHQV7RDLCZUHY2XMXUAD2PX4ZBLWKTM47N6PH7UOGBRXPMWVOA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.853517822 (151h11m40.549366575s) [noise=true , address=145.79.227.22:4509 , piece_id=7RA5NJ72PI3DT4IC2ECVLZ3A2HXXL546ELVDXHZW4OXNN4D36OEA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.855547508 (151h11m40.551396261s) [noise=true , address=145.79.227.9:4366 , piece_id=7ZRECKBQC4S6QLERBZY442WYS4H35XWOZYSGGPHSU7NQ6WL4UNVA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.856103106 (151h11m40.551951859s) [noise=true , address=145.79.227.25:4618 , piece_id=XRUPY5BUPG7BXN2MI6MGYO4N3GW6TZDJE46MZFNRELWANHHG3N5A, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.860417321 (151h11m40.556266064s) [noise=true , address=145.79.227.8:4330 , piece_id=4A47HIUX2MTJ2673XZZCMMF4T235TS4JWMNYJEVHHCKAHQX6APMA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.865602796 (151h11m40.561451549s) [noise=true , address=145.79.227.23:4547 , piece_id=K54H437KW5AAYUWOTMDBDP5GJ3ZABFKLAAOSS6WS5G5CROYEHODQ, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.879393757 (151h11m40.57524251s) [noise=true , address=145.79.227.31:4797 , piece_id=2TFSZNURIAU573QUA5DUS7AJZNY2OXC6TJL3ST4VXIWY2WVB5YJA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.905541449 (151h11m40.601390202s) [noise=true , address=145.79.227.26:4654 , piece_id=E5J22KNPQSETSY443QQZXFFKUUJFHWFV22SQAVZ4BFQ4M6YWCNQQ, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.929018836 (151h11m40.624867589s) [noise=true , address=145.79.227.24:4581 , piece_id=TSCVCH2KJEVMD4AVOEIHGTMRCRZ6NZMZA2CTQ6FUO2EIL3JAVYEA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:35.899554917 (151h12m7.59540368s) [noise=true , address=77.90.31.17:4771 , piece_id=KVFE7VYW3RZJ25BWSH4BIRRQCMWDYOIK7EJ62XVREGN26YN7XYMQ, seg_pos={0 0}, upload=133242]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: write tcp 188.166.225.170:60676->77.90.31.17:4771: use of closed network connection; write tcp 188.166.225.170:60676->77.90.31.17:4771: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60676->77.90.31.17:4771: use of closed network connection Nov 5 18:52:35.899727506 (151h12m7.595576259s) [noise=true , address=77.90.31.13:4628 , piece_id=QFTEHZ5FY7A5DPZZ44TRXGFRHPRNJZJHLQZDGRGOWYFEZYIZT2NQ, seg_pos={0 0}, upload=133242]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: write tcp 188.166.225.170:42026->77.90.31.13:4628: use of closed network connection; write tcp 188.166.225.170:42026->77.90.31.13:4628: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42026->77.90.31.13:4628: use of closed network connection Nov 5 18:52:35.899860439 (151h12m7.595709192s) [noise=true , address=45.91.116.7:4010 , piece_id=EBVGGNQF3N4MTUIATJUWK4QZN7SEOWI6SIAW6N7JZOB5G6H33NEQ, seg_pos={0 0}, upload=133242]: Done uploading piece. err: ecclient: upload failed (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, address:45.91.116.7:4010): protocol: write tcp 188.166.225.170:40036->45.91.116.7:4010: use of closed network connection; write tcp 188.166.225.170:40036->45.91.116.7:4010: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40036->45.91.116.7:4010: use of closed network connection Nov 5 18:52:35.899923829 (151h12m7.595772572s) [noise=true , address=77.90.31.18:4805 , piece_id=U2U34KK4AJK4A5HGNTHEYCIARLAR4X5NLOVCF63TN3PJUIRMMIHA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: write tcp 188.166.225.170:60758->77.90.31.18:4805: use of closed network connection; write tcp 188.166.225.170:60758->77.90.31.18:4805: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60758->77.90.31.18:4805: use of closed network connection Nov 5 18:52:35.900052084 (151h12m7.595900837s) [noise=true , address=77.90.31.16:4734 , piece_id=HFWVFM46PZFLZT6YNZVIA6HKLVN2PTV2GETZEKX356DMDGQO4HOA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: write tcp 188.166.225.170:50728->77.90.31.16:4734: use of closed network connection; write tcp 188.166.225.170:50728->77.90.31.16:4734: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50728->77.90.31.16:4734: use of closed network connection Nov 5 18:52:35.900231175 (151h12m7.596079938s) [noise=true , address=45.91.116.11:4117 , piece_id=QUGUYHEJWEEV733CBC3ONMV7XWDKVH2OFLQWXPQHPKZ7GTH4RLUA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: ecclient: upload failed (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, address:45.91.116.11:4117): protocol: write tcp 188.166.225.170:41292->45.91.116.11:4117: use of closed network connection; write tcp 188.166.225.170:41292->45.91.116.11:4117: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41292->45.91.116.11:4117: use of closed network connection Nov 5 18:52:35.900342116 (151h12m7.596190859s) [noise=true , address=77.90.31.15:4698 , piece_id=7VACL26N7ZCBXIA2I3XL5Q3UJ2BAHDIPXX4VZCQX3S63ABNW625Q, seg_pos={0 0}, upload=133242]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: write tcp 188.166.225.170:41118->77.90.31.15:4698: use of closed network connection; write tcp 188.166.225.170:41118->77.90.31.15:4698: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41118->77.90.31.15:4698: use of closed network connection Nov 5 18:52:35.900382814 (151h12m7.596231567s) [seg_pos={0 0}, upload=133242]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Nov 5 18:52:36.162442033 (151h12m7.858290786s) [upload=133242]: Done uploading segments... Nov 5 18:52:36.162481568 (151h12m7.858330321s) [upload=133242]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 5.95313009s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [.....................XX...X.................................] duration: 3.509138897s trace: https://jaeger.dev.storj.io/trace/204ab2570dc82d9c FAIL (2025-11-05 19:18:38 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.509186798s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....................X...X..................................] duration: 6.251092637s trace: https://jaeger.dev.storj.io/trace/74cf1e30f6d49950 FAIL (2025-11-05 19:18:08 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.251123063s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [....................X..X....................................] duration: 5.743257077s trace: https://jaeger.dev.storj.io/trace/3e2ae8884260b157 FAIL (2025-11-05 19:18:08 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.743293396s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [....................X...X...................................] duration: 5.801207444s trace: https://jaeger.dev.storj.io/trace/2e576aae48e8f03e FAIL (2025-11-05 19:17:58 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== 5 18:52:06.850167887 (151h11m38.54601663s) [noise=true , address=145.79.227.25:4618 , piece_id=XRUPY5BUPG7BXN2MI6MGYO4N3GW6TZDJE46MZFNRELWANHHG3N5A, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.850747301 (151h11m38.546596054s) [noise=true , address=77.90.31.15:4698 , piece_id=7VACL26N7ZCBXIA2I3XL5Q3UJ2BAHDIPXX4VZCQX3S63ABNW625Q, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.851222605 (151h11m38.547071358s) [noise=true , address=145.79.227.17:4972 , piece_id=H7UHQV7RDLCZUHY2XMXUAD2PX4ZBLWKTM47N6PH7UOGBRXPMWVOA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.851558575 (151h11m38.547407319s) [noise=true , address=145.79.227.22:4509 , piece_id=7RA5NJ72PI3DT4IC2ECVLZ3A2HXXL546ELVDXHZW4OXNN4D36OEA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.851909263 (151h11m38.547758006s) [noise=true , address=145.79.227.26:4654 , piece_id=E5J22KNPQSETSY443QQZXFFKUUJFHWFV22SQAVZ4BFQ4M6YWCNQQ, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.852327540 (151h11m38.548176293s) [noise=true , address=45.91.116.7:4010 , piece_id=EBVGGNQF3N4MTUIATJUWK4QZN7SEOWI6SIAW6N7JZOB5G6H33NEQ, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.852728624 (151h11m38.548577377s) [noise=true , address=145.79.227.9:4366 , piece_id=7ZRECKBQC4S6QLERBZY442WYS4H35XWOZYSGGPHSU7NQ6WL4UNVA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.853978774 (151h11m38.549827528s) [noise=true , address=77.90.31.18:4805 , piece_id=U2U34KK4AJK4A5HGNTHEYCIARLAR4X5NLOVCF63TN3PJUIRMMIHA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.854447296 (151h11m38.550296059s) [noise=true , address=77.90.31.16:4734 , piece_id=HFWVFM46PZFLZT6YNZVIA6HKLVN2PTV2GETZEKX356DMDGQO4HOA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.856523089 (151h11m38.552371842s) [noise=true , address=45.91.116.11:4117 , piece_id=QUGUYHEJWEEV733CBC3ONMV7XWDKVH2OFLQWXPQHPKZ7GTH4RLUA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.859490379 (151h11m38.555339122s) [noise=true , address=77.90.31.13:4628 , piece_id=QFTEHZ5FY7A5DPZZ44TRXGFRHPRNJZJHLQZDGRGOWYFEZYIZT2NQ, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.861350291 (151h11m38.557199034s) [noise=true , address=145.79.227.31:4797 , piece_id=2TFSZNURIAU573QUA5DUS7AJZNY2OXC6TJL3ST4VXIWY2WVB5YJA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.863368665 (151h11m38.559217418s) [noise=true , address=145.79.227.23:4547 , piece_id=K54H437KW5AAYUWOTMDBDP5GJ3ZABFKLAAOSS6WS5G5CROYEHODQ, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.864848794 (151h11m38.560697547s) [noise=true , address=145.79.227.24:4581 , piece_id=TSCVCH2KJEVMD4AVOEIHGTMRCRZ6NZMZA2CTQ6FUO2EIL3JAVYEA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.865644930 (151h11m38.561493683s) [noise=true , address=145.79.227.8:4330 , piece_id=4A47HIUX2MTJ2673XZZCMMF4T235TS4JWMNYJEVHHCKAHQX6APMA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:08.560504137 (151h11m40.25635289s) [upload=133242]: Next returned nil segment Nov 5 18:52:08.560522693 (151h11m40.256371446s) [upload=133242]: Waiting for error group managing segments... Nov 5 18:52:08.850333179 (151h11m40.546181932s) [noise=true , address=145.79.227.17:4972 , piece_id=H7UHQV7RDLCZUHY2XMXUAD2PX4ZBLWKTM47N6PH7UOGBRXPMWVOA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.853517822 (151h11m40.549366575s) [noise=true , address=145.79.227.22:4509 , piece_id=7RA5NJ72PI3DT4IC2ECVLZ3A2HXXL546ELVDXHZW4OXNN4D36OEA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.855547508 (151h11m40.551396261s) [noise=true , address=145.79.227.9:4366 , piece_id=7ZRECKBQC4S6QLERBZY442WYS4H35XWOZYSGGPHSU7NQ6WL4UNVA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.856103106 (151h11m40.551951859s) [noise=true , address=145.79.227.25:4618 , piece_id=XRUPY5BUPG7BXN2MI6MGYO4N3GW6TZDJE46MZFNRELWANHHG3N5A, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.860417321 (151h11m40.556266064s) [noise=true , address=145.79.227.8:4330 , piece_id=4A47HIUX2MTJ2673XZZCMMF4T235TS4JWMNYJEVHHCKAHQX6APMA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.865602796 (151h11m40.561451549s) [noise=true , address=145.79.227.23:4547 , piece_id=K54H437KW5AAYUWOTMDBDP5GJ3ZABFKLAAOSS6WS5G5CROYEHODQ, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.879393757 (151h11m40.57524251s) [noise=true , address=145.79.227.31:4797 , piece_id=2TFSZNURIAU573QUA5DUS7AJZNY2OXC6TJL3ST4VXIWY2WVB5YJA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.905541449 (151h11m40.601390202s) [noise=true , address=145.79.227.26:4654 , piece_id=E5J22KNPQSETSY443QQZXFFKUUJFHWFV22SQAVZ4BFQ4M6YWCNQQ, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.929018836 (151h11m40.624867589s) [noise=true , address=145.79.227.24:4581 , piece_id=TSCVCH2KJEVMD4AVOEIHGTMRCRZ6NZMZA2CTQ6FUO2EIL3JAVYEA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:35.899554917 (151h12m7.59540368s) [noise=true , address=77.90.31.17:4771 , piece_id=KVFE7VYW3RZJ25BWSH4BIRRQCMWDYOIK7EJ62XVREGN26YN7XYMQ, seg_pos={0 0}, upload=133242]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: write tcp 188.166.225.170:60676->77.90.31.17:4771: use of closed network connection; write tcp 188.166.225.170:60676->77.90.31.17:4771: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60676->77.90.31.17:4771: use of closed network connection Nov 5 18:52:35.899727506 (151h12m7.595576259s) [noise=true , address=77.90.31.13:4628 , piece_id=QFTEHZ5FY7A5DPZZ44TRXGFRHPRNJZJHLQZDGRGOWYFEZYIZT2NQ, seg_pos={0 0}, upload=133242]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: write tcp 188.166.225.170:42026->77.90.31.13:4628: use of closed network connection; write tcp 188.166.225.170:42026->77.90.31.13:4628: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42026->77.90.31.13:4628: use of closed network connection Nov 5 18:52:35.899860439 (151h12m7.595709192s) [noise=true , address=45.91.116.7:4010 , piece_id=EBVGGNQF3N4MTUIATJUWK4QZN7SEOWI6SIAW6N7JZOB5G6H33NEQ, seg_pos={0 0}, upload=133242]: Done uploading piece. err: ecclient: upload failed (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, address:45.91.116.7:4010): protocol: write tcp 188.166.225.170:40036->45.91.116.7:4010: use of closed network connection; write tcp 188.166.225.170:40036->45.91.116.7:4010: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40036->45.91.116.7:4010: use of closed network connection Nov 5 18:52:35.899923829 (151h12m7.595772572s) [noise=true , address=77.90.31.18:4805 , piece_id=U2U34KK4AJK4A5HGNTHEYCIARLAR4X5NLOVCF63TN3PJUIRMMIHA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: write tcp 188.166.225.170:60758->77.90.31.18:4805: use of closed network connection; write tcp 188.166.225.170:60758->77.90.31.18:4805: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60758->77.90.31.18:4805: use of closed network connection Nov 5 18:52:35.900052084 (151h12m7.595900837s) [noise=true , address=77.90.31.16:4734 , piece_id=HFWVFM46PZFLZT6YNZVIA6HKLVN2PTV2GETZEKX356DMDGQO4HOA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: write tcp 188.166.225.170:50728->77.90.31.16:4734: use of closed network connection; write tcp 188.166.225.170:50728->77.90.31.16:4734: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50728->77.90.31.16:4734: use of closed network connection Nov 5 18:52:35.900231175 (151h12m7.596079938s) [noise=true , address=45.91.116.11:4117 , piece_id=QUGUYHEJWEEV733CBC3ONMV7XWDKVH2OFLQWXPQHPKZ7GTH4RLUA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: ecclient: upload failed (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, address:45.91.116.11:4117): protocol: write tcp 188.166.225.170:41292->45.91.116.11:4117: use of closed network connection; write tcp 188.166.225.170:41292->45.91.116.11:4117: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41292->45.91.116.11:4117: use of closed network connection Nov 5 18:52:35.900342116 (151h12m7.596190859s) [noise=true , address=77.90.31.15:4698 , piece_id=7VACL26N7ZCBXIA2I3XL5Q3UJ2BAHDIPXX4VZCQX3S63ABNW625Q, seg_pos={0 0}, upload=133242]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: write tcp 188.166.225.170:41118->77.90.31.15:4698: use of closed network connection; write tcp 188.166.225.170:41118->77.90.31.15:4698: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41118->77.90.31.15:4698: use of closed network connection Nov 5 18:52:35.900382814 (151h12m7.596231567s) [seg_pos={0 0}, upload=133242]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Nov 5 18:52:36.162442033 (151h12m7.858290786s) [upload=133242]: Done uploading segments... Nov 5 18:52:36.162481568 (151h12m7.858330321s) [upload=133242]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 5.801234206s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [......................XX...X................................] duration: 3.437505646s trace: https://jaeger.dev.storj.io/trace/204ab2570dc82d9c FAIL (2025-11-05 19:17:43 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.437536856s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [......................X...X.................................] duration: 6.271009283s trace: https://jaeger.dev.storj.io/trace/74cf1e30f6d49950 FAIL (2025-11-05 19:17:37 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.271034331s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....................X...X..................................] duration: 5.8945676s trace: https://jaeger.dev.storj.io/trace/2e576aae48e8f03e FAIL (2025-11-05 19:17:35 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== 5 18:52:06.850167887 (151h11m38.54601663s) [noise=true , address=145.79.227.25:4618 , piece_id=XRUPY5BUPG7BXN2MI6MGYO4N3GW6TZDJE46MZFNRELWANHHG3N5A, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.850747301 (151h11m38.546596054s) [noise=true , address=77.90.31.15:4698 , piece_id=7VACL26N7ZCBXIA2I3XL5Q3UJ2BAHDIPXX4VZCQX3S63ABNW625Q, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.851222605 (151h11m38.547071358s) [noise=true , address=145.79.227.17:4972 , piece_id=H7UHQV7RDLCZUHY2XMXUAD2PX4ZBLWKTM47N6PH7UOGBRXPMWVOA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.851558575 (151h11m38.547407319s) [noise=true , address=145.79.227.22:4509 , piece_id=7RA5NJ72PI3DT4IC2ECVLZ3A2HXXL546ELVDXHZW4OXNN4D36OEA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.851909263 (151h11m38.547758006s) [noise=true , address=145.79.227.26:4654 , piece_id=E5J22KNPQSETSY443QQZXFFKUUJFHWFV22SQAVZ4BFQ4M6YWCNQQ, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.852327540 (151h11m38.548176293s) [noise=true , address=45.91.116.7:4010 , piece_id=EBVGGNQF3N4MTUIATJUWK4QZN7SEOWI6SIAW6N7JZOB5G6H33NEQ, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.852728624 (151h11m38.548577377s) [noise=true , address=145.79.227.9:4366 , piece_id=7ZRECKBQC4S6QLERBZY442WYS4H35XWOZYSGGPHSU7NQ6WL4UNVA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.853978774 (151h11m38.549827528s) [noise=true , address=77.90.31.18:4805 , piece_id=U2U34KK4AJK4A5HGNTHEYCIARLAR4X5NLOVCF63TN3PJUIRMMIHA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.854447296 (151h11m38.550296059s) [noise=true , address=77.90.31.16:4734 , piece_id=HFWVFM46PZFLZT6YNZVIA6HKLVN2PTV2GETZEKX356DMDGQO4HOA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.856523089 (151h11m38.552371842s) [noise=true , address=45.91.116.11:4117 , piece_id=QUGUYHEJWEEV733CBC3ONMV7XWDKVH2OFLQWXPQHPKZ7GTH4RLUA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.859490379 (151h11m38.555339122s) [noise=true , address=77.90.31.13:4628 , piece_id=QFTEHZ5FY7A5DPZZ44TRXGFRHPRNJZJHLQZDGRGOWYFEZYIZT2NQ, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.861350291 (151h11m38.557199034s) [noise=true , address=145.79.227.31:4797 , piece_id=2TFSZNURIAU573QUA5DUS7AJZNY2OXC6TJL3ST4VXIWY2WVB5YJA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.863368665 (151h11m38.559217418s) [noise=true , address=145.79.227.23:4547 , piece_id=K54H437KW5AAYUWOTMDBDP5GJ3ZABFKLAAOSS6WS5G5CROYEHODQ, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.864848794 (151h11m38.560697547s) [noise=true , address=145.79.227.24:4581 , piece_id=TSCVCH2KJEVMD4AVOEIHGTMRCRZ6NZMZA2CTQ6FUO2EIL3JAVYEA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.865644930 (151h11m38.561493683s) [noise=true , address=145.79.227.8:4330 , piece_id=4A47HIUX2MTJ2673XZZCMMF4T235TS4JWMNYJEVHHCKAHQX6APMA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:08.560504137 (151h11m40.25635289s) [upload=133242]: Next returned nil segment Nov 5 18:52:08.560522693 (151h11m40.256371446s) [upload=133242]: Waiting for error group managing segments... Nov 5 18:52:08.850333179 (151h11m40.546181932s) [noise=true , address=145.79.227.17:4972 , piece_id=H7UHQV7RDLCZUHY2XMXUAD2PX4ZBLWKTM47N6PH7UOGBRXPMWVOA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.853517822 (151h11m40.549366575s) [noise=true , address=145.79.227.22:4509 , piece_id=7RA5NJ72PI3DT4IC2ECVLZ3A2HXXL546ELVDXHZW4OXNN4D36OEA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.855547508 (151h11m40.551396261s) [noise=true , address=145.79.227.9:4366 , piece_id=7ZRECKBQC4S6QLERBZY442WYS4H35XWOZYSGGPHSU7NQ6WL4UNVA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.856103106 (151h11m40.551951859s) [noise=true , address=145.79.227.25:4618 , piece_id=XRUPY5BUPG7BXN2MI6MGYO4N3GW6TZDJE46MZFNRELWANHHG3N5A, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.860417321 (151h11m40.556266064s) [noise=true , address=145.79.227.8:4330 , piece_id=4A47HIUX2MTJ2673XZZCMMF4T235TS4JWMNYJEVHHCKAHQX6APMA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.865602796 (151h11m40.561451549s) [noise=true , address=145.79.227.23:4547 , piece_id=K54H437KW5AAYUWOTMDBDP5GJ3ZABFKLAAOSS6WS5G5CROYEHODQ, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.879393757 (151h11m40.57524251s) [noise=true , address=145.79.227.31:4797 , piece_id=2TFSZNURIAU573QUA5DUS7AJZNY2OXC6TJL3ST4VXIWY2WVB5YJA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.905541449 (151h11m40.601390202s) [noise=true , address=145.79.227.26:4654 , piece_id=E5J22KNPQSETSY443QQZXFFKUUJFHWFV22SQAVZ4BFQ4M6YWCNQQ, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.929018836 (151h11m40.624867589s) [noise=true , address=145.79.227.24:4581 , piece_id=TSCVCH2KJEVMD4AVOEIHGTMRCRZ6NZMZA2CTQ6FUO2EIL3JAVYEA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:35.899554917 (151h12m7.59540368s) [noise=true , address=77.90.31.17:4771 , piece_id=KVFE7VYW3RZJ25BWSH4BIRRQCMWDYOIK7EJ62XVREGN26YN7XYMQ, seg_pos={0 0}, upload=133242]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: write tcp 188.166.225.170:60676->77.90.31.17:4771: use of closed network connection; write tcp 188.166.225.170:60676->77.90.31.17:4771: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60676->77.90.31.17:4771: use of closed network connection Nov 5 18:52:35.899727506 (151h12m7.595576259s) [noise=true , address=77.90.31.13:4628 , piece_id=QFTEHZ5FY7A5DPZZ44TRXGFRHPRNJZJHLQZDGRGOWYFEZYIZT2NQ, seg_pos={0 0}, upload=133242]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: write tcp 188.166.225.170:42026->77.90.31.13:4628: use of closed network connection; write tcp 188.166.225.170:42026->77.90.31.13:4628: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42026->77.90.31.13:4628: use of closed network connection Nov 5 18:52:35.899860439 (151h12m7.595709192s) [noise=true , address=45.91.116.7:4010 , piece_id=EBVGGNQF3N4MTUIATJUWK4QZN7SEOWI6SIAW6N7JZOB5G6H33NEQ, seg_pos={0 0}, upload=133242]: Done uploading piece. err: ecclient: upload failed (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, address:45.91.116.7:4010): protocol: write tcp 188.166.225.170:40036->45.91.116.7:4010: use of closed network connection; write tcp 188.166.225.170:40036->45.91.116.7:4010: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40036->45.91.116.7:4010: use of closed network connection Nov 5 18:52:35.899923829 (151h12m7.595772572s) [noise=true , address=77.90.31.18:4805 , piece_id=U2U34KK4AJK4A5HGNTHEYCIARLAR4X5NLOVCF63TN3PJUIRMMIHA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: write tcp 188.166.225.170:60758->77.90.31.18:4805: use of closed network connection; write tcp 188.166.225.170:60758->77.90.31.18:4805: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60758->77.90.31.18:4805: use of closed network connection Nov 5 18:52:35.900052084 (151h12m7.595900837s) [noise=true , address=77.90.31.16:4734 , piece_id=HFWVFM46PZFLZT6YNZVIA6HKLVN2PTV2GETZEKX356DMDGQO4HOA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: write tcp 188.166.225.170:50728->77.90.31.16:4734: use of closed network connection; write tcp 188.166.225.170:50728->77.90.31.16:4734: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50728->77.90.31.16:4734: use of closed network connection Nov 5 18:52:35.900231175 (151h12m7.596079938s) [noise=true , address=45.91.116.11:4117 , piece_id=QUGUYHEJWEEV733CBC3ONMV7XWDKVH2OFLQWXPQHPKZ7GTH4RLUA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: ecclient: upload failed (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, address:45.91.116.11:4117): protocol: write tcp 188.166.225.170:41292->45.91.116.11:4117: use of closed network connection; write tcp 188.166.225.170:41292->45.91.116.11:4117: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41292->45.91.116.11:4117: use of closed network connection Nov 5 18:52:35.900342116 (151h12m7.596190859s) [noise=true , address=77.90.31.15:4698 , piece_id=7VACL26N7ZCBXIA2I3XL5Q3UJ2BAHDIPXX4VZCQX3S63ABNW625Q, seg_pos={0 0}, upload=133242]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: write tcp 188.166.225.170:41118->77.90.31.15:4698: use of closed network connection; write tcp 188.166.225.170:41118->77.90.31.15:4698: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41118->77.90.31.15:4698: use of closed network connection Nov 5 18:52:35.900382814 (151h12m7.596231567s) [seg_pos={0 0}, upload=133242]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Nov 5 18:52:36.162442033 (151h12m7.858290786s) [upload=133242]: Done uploading segments... Nov 5 18:52:36.162481568 (151h12m7.858330321s) [upload=133242]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 5.894597699s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.....................X..X...................................] duration: 5.718571336s trace: https://jaeger.dev.storj.io/trace/3e2ae8884260b157 FAIL (2025-11-05 19:17:32 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.718598678s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [.......................XX...X...............................] duration: 3.506712472s trace: https://jaeger.dev.storj.io/trace/204ab2570dc82d9c FAIL (2025-11-05 19:17:19 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.506788066s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [......................X..X..................................] duration: 5.769047476s trace: https://jaeger.dev.storj.io/trace/3e2ae8884260b157 FAIL (2025-11-05 19:17:07 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.769378615s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......................X...X................................] duration: 6.206544847s trace: https://jaeger.dev.storj.io/trace/74cf1e30f6d49950 FAIL (2025-11-05 19:17:04 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.206571088s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [........................XX...X..............................] duration: 3.468622322s trace: https://jaeger.dev.storj.io/trace/204ab2570dc82d9c FAIL (2025-11-05 19:16:52 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.468652329s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [......................X...X.................................] duration: 8.707911089s trace: https://jaeger.dev.storj.io/trace/2e576aae48e8f03e FAIL (2025-11-05 19:16:39 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== 5 18:52:06.850167887 (151h11m38.54601663s) [noise=true , address=145.79.227.25:4618 , piece_id=XRUPY5BUPG7BXN2MI6MGYO4N3GW6TZDJE46MZFNRELWANHHG3N5A, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.850747301 (151h11m38.546596054s) [noise=true , address=77.90.31.15:4698 , piece_id=7VACL26N7ZCBXIA2I3XL5Q3UJ2BAHDIPXX4VZCQX3S63ABNW625Q, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.851222605 (151h11m38.547071358s) [noise=true , address=145.79.227.17:4972 , piece_id=H7UHQV7RDLCZUHY2XMXUAD2PX4ZBLWKTM47N6PH7UOGBRXPMWVOA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.851558575 (151h11m38.547407319s) [noise=true , address=145.79.227.22:4509 , piece_id=7RA5NJ72PI3DT4IC2ECVLZ3A2HXXL546ELVDXHZW4OXNN4D36OEA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.851909263 (151h11m38.547758006s) [noise=true , address=145.79.227.26:4654 , piece_id=E5J22KNPQSETSY443QQZXFFKUUJFHWFV22SQAVZ4BFQ4M6YWCNQQ, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.852327540 (151h11m38.548176293s) [noise=true , address=45.91.116.7:4010 , piece_id=EBVGGNQF3N4MTUIATJUWK4QZN7SEOWI6SIAW6N7JZOB5G6H33NEQ, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.852728624 (151h11m38.548577377s) [noise=true , address=145.79.227.9:4366 , piece_id=7ZRECKBQC4S6QLERBZY442WYS4H35XWOZYSGGPHSU7NQ6WL4UNVA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.853978774 (151h11m38.549827528s) [noise=true , address=77.90.31.18:4805 , piece_id=U2U34KK4AJK4A5HGNTHEYCIARLAR4X5NLOVCF63TN3PJUIRMMIHA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.854447296 (151h11m38.550296059s) [noise=true , address=77.90.31.16:4734 , piece_id=HFWVFM46PZFLZT6YNZVIA6HKLVN2PTV2GETZEKX356DMDGQO4HOA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.856523089 (151h11m38.552371842s) [noise=true , address=45.91.116.11:4117 , piece_id=QUGUYHEJWEEV733CBC3ONMV7XWDKVH2OFLQWXPQHPKZ7GTH4RLUA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.859490379 (151h11m38.555339122s) [noise=true , address=77.90.31.13:4628 , piece_id=QFTEHZ5FY7A5DPZZ44TRXGFRHPRNJZJHLQZDGRGOWYFEZYIZT2NQ, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.861350291 (151h11m38.557199034s) [noise=true , address=145.79.227.31:4797 , piece_id=2TFSZNURIAU573QUA5DUS7AJZNY2OXC6TJL3ST4VXIWY2WVB5YJA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.863368665 (151h11m38.559217418s) [noise=true , address=145.79.227.23:4547 , piece_id=K54H437KW5AAYUWOTMDBDP5GJ3ZABFKLAAOSS6WS5G5CROYEHODQ, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.864848794 (151h11m38.560697547s) [noise=true , address=145.79.227.24:4581 , piece_id=TSCVCH2KJEVMD4AVOEIHGTMRCRZ6NZMZA2CTQ6FUO2EIL3JAVYEA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.865644930 (151h11m38.561493683s) [noise=true , address=145.79.227.8:4330 , piece_id=4A47HIUX2MTJ2673XZZCMMF4T235TS4JWMNYJEVHHCKAHQX6APMA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:08.560504137 (151h11m40.25635289s) [upload=133242]: Next returned nil segment Nov 5 18:52:08.560522693 (151h11m40.256371446s) [upload=133242]: Waiting for error group managing segments... Nov 5 18:52:08.850333179 (151h11m40.546181932s) [noise=true , address=145.79.227.17:4972 , piece_id=H7UHQV7RDLCZUHY2XMXUAD2PX4ZBLWKTM47N6PH7UOGBRXPMWVOA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.853517822 (151h11m40.549366575s) [noise=true , address=145.79.227.22:4509 , piece_id=7RA5NJ72PI3DT4IC2ECVLZ3A2HXXL546ELVDXHZW4OXNN4D36OEA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.855547508 (151h11m40.551396261s) [noise=true , address=145.79.227.9:4366 , piece_id=7ZRECKBQC4S6QLERBZY442WYS4H35XWOZYSGGPHSU7NQ6WL4UNVA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.856103106 (151h11m40.551951859s) [noise=true , address=145.79.227.25:4618 , piece_id=XRUPY5BUPG7BXN2MI6MGYO4N3GW6TZDJE46MZFNRELWANHHG3N5A, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.860417321 (151h11m40.556266064s) [noise=true , address=145.79.227.8:4330 , piece_id=4A47HIUX2MTJ2673XZZCMMF4T235TS4JWMNYJEVHHCKAHQX6APMA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.865602796 (151h11m40.561451549s) [noise=true , address=145.79.227.23:4547 , piece_id=K54H437KW5AAYUWOTMDBDP5GJ3ZABFKLAAOSS6WS5G5CROYEHODQ, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.879393757 (151h11m40.57524251s) [noise=true , address=145.79.227.31:4797 , piece_id=2TFSZNURIAU573QUA5DUS7AJZNY2OXC6TJL3ST4VXIWY2WVB5YJA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.905541449 (151h11m40.601390202s) [noise=true , address=145.79.227.26:4654 , piece_id=E5J22KNPQSETSY443QQZXFFKUUJFHWFV22SQAVZ4BFQ4M6YWCNQQ, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.929018836 (151h11m40.624867589s) [noise=true , address=145.79.227.24:4581 , piece_id=TSCVCH2KJEVMD4AVOEIHGTMRCRZ6NZMZA2CTQ6FUO2EIL3JAVYEA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:35.899554917 (151h12m7.59540368s) [noise=true , address=77.90.31.17:4771 , piece_id=KVFE7VYW3RZJ25BWSH4BIRRQCMWDYOIK7EJ62XVREGN26YN7XYMQ, seg_pos={0 0}, upload=133242]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: write tcp 188.166.225.170:60676->77.90.31.17:4771: use of closed network connection; write tcp 188.166.225.170:60676->77.90.31.17:4771: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60676->77.90.31.17:4771: use of closed network connection Nov 5 18:52:35.899727506 (151h12m7.595576259s) [noise=true , address=77.90.31.13:4628 , piece_id=QFTEHZ5FY7A5DPZZ44TRXGFRHPRNJZJHLQZDGRGOWYFEZYIZT2NQ, seg_pos={0 0}, upload=133242]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: write tcp 188.166.225.170:42026->77.90.31.13:4628: use of closed network connection; write tcp 188.166.225.170:42026->77.90.31.13:4628: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42026->77.90.31.13:4628: use of closed network connection Nov 5 18:52:35.899860439 (151h12m7.595709192s) [noise=true , address=45.91.116.7:4010 , piece_id=EBVGGNQF3N4MTUIATJUWK4QZN7SEOWI6SIAW6N7JZOB5G6H33NEQ, seg_pos={0 0}, upload=133242]: Done uploading piece. err: ecclient: upload failed (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, address:45.91.116.7:4010): protocol: write tcp 188.166.225.170:40036->45.91.116.7:4010: use of closed network connection; write tcp 188.166.225.170:40036->45.91.116.7:4010: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40036->45.91.116.7:4010: use of closed network connection Nov 5 18:52:35.899923829 (151h12m7.595772572s) [noise=true , address=77.90.31.18:4805 , piece_id=U2U34KK4AJK4A5HGNTHEYCIARLAR4X5NLOVCF63TN3PJUIRMMIHA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: write tcp 188.166.225.170:60758->77.90.31.18:4805: use of closed network connection; write tcp 188.166.225.170:60758->77.90.31.18:4805: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60758->77.90.31.18:4805: use of closed network connection Nov 5 18:52:35.900052084 (151h12m7.595900837s) [noise=true , address=77.90.31.16:4734 , piece_id=HFWVFM46PZFLZT6YNZVIA6HKLVN2PTV2GETZEKX356DMDGQO4HOA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: write tcp 188.166.225.170:50728->77.90.31.16:4734: use of closed network connection; write tcp 188.166.225.170:50728->77.90.31.16:4734: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50728->77.90.31.16:4734: use of closed network connection Nov 5 18:52:35.900231175 (151h12m7.596079938s) [noise=true , address=45.91.116.11:4117 , piece_id=QUGUYHEJWEEV733CBC3ONMV7XWDKVH2OFLQWXPQHPKZ7GTH4RLUA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: ecclient: upload failed (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, address:45.91.116.11:4117): protocol: write tcp 188.166.225.170:41292->45.91.116.11:4117: use of closed network connection; write tcp 188.166.225.170:41292->45.91.116.11:4117: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41292->45.91.116.11:4117: use of closed network connection Nov 5 18:52:35.900342116 (151h12m7.596190859s) [noise=true , address=77.90.31.15:4698 , piece_id=7VACL26N7ZCBXIA2I3XL5Q3UJ2BAHDIPXX4VZCQX3S63ABNW625Q, seg_pos={0 0}, upload=133242]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: write tcp 188.166.225.170:41118->77.90.31.15:4698: use of closed network connection; write tcp 188.166.225.170:41118->77.90.31.15:4698: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41118->77.90.31.15:4698: use of closed network connection Nov 5 18:52:35.900382814 (151h12m7.596231567s) [seg_pos={0 0}, upload=133242]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Nov 5 18:52:36.162442033 (151h12m7.858290786s) [upload=133242]: Done uploading segments... Nov 5 18:52:36.162481568 (151h12m7.858330321s) [upload=133242]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 8.707940845s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [........................X...X...............................] duration: 6.232880639s trace: https://jaeger.dev.storj.io/trace/74cf1e30f6d49950 FAIL (2025-11-05 19:16:27 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.232911187s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [.........................XX...X.............................] duration: 3.463051364s trace: https://jaeger.dev.storj.io/trace/204ab2570dc82d9c FAIL (2025-11-05 19:16:23 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.463085919s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.......................X..X.................................] duration: 5.727483027s trace: https://jaeger.dev.storj.io/trace/3e2ae8884260b157 FAIL (2025-11-05 19:16:05 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.727513094s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.........................X...X..............................] duration: 6.194396513s trace: https://jaeger.dev.storj.io/trace/74cf1e30f6d49950 FAIL (2025-11-05 19:15:54 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.194456586s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [..........................XX...X............................] duration: 3.491800337s trace: https://jaeger.dev.storj.io/trace/204ab2570dc82d9c FAIL (2025-11-05 19:15:53 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.491827869s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......................X...X................................] duration: 5.96266913s trace: https://jaeger.dev.storj.io/trace/2e576aae48e8f03e FAIL (2025-11-05 19:15:28 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== 5 18:52:06.850167887 (151h11m38.54601663s) [noise=true , address=145.79.227.25:4618 , piece_id=XRUPY5BUPG7BXN2MI6MGYO4N3GW6TZDJE46MZFNRELWANHHG3N5A, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.850747301 (151h11m38.546596054s) [noise=true , address=77.90.31.15:4698 , piece_id=7VACL26N7ZCBXIA2I3XL5Q3UJ2BAHDIPXX4VZCQX3S63ABNW625Q, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.851222605 (151h11m38.547071358s) [noise=true , address=145.79.227.17:4972 , piece_id=H7UHQV7RDLCZUHY2XMXUAD2PX4ZBLWKTM47N6PH7UOGBRXPMWVOA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.851558575 (151h11m38.547407319s) [noise=true , address=145.79.227.22:4509 , piece_id=7RA5NJ72PI3DT4IC2ECVLZ3A2HXXL546ELVDXHZW4OXNN4D36OEA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.851909263 (151h11m38.547758006s) [noise=true , address=145.79.227.26:4654 , piece_id=E5J22KNPQSETSY443QQZXFFKUUJFHWFV22SQAVZ4BFQ4M6YWCNQQ, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.852327540 (151h11m38.548176293s) [noise=true , address=45.91.116.7:4010 , piece_id=EBVGGNQF3N4MTUIATJUWK4QZN7SEOWI6SIAW6N7JZOB5G6H33NEQ, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.852728624 (151h11m38.548577377s) [noise=true , address=145.79.227.9:4366 , piece_id=7ZRECKBQC4S6QLERBZY442WYS4H35XWOZYSGGPHSU7NQ6WL4UNVA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.853978774 (151h11m38.549827528s) [noise=true , address=77.90.31.18:4805 , piece_id=U2U34KK4AJK4A5HGNTHEYCIARLAR4X5NLOVCF63TN3PJUIRMMIHA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.854447296 (151h11m38.550296059s) [noise=true , address=77.90.31.16:4734 , piece_id=HFWVFM46PZFLZT6YNZVIA6HKLVN2PTV2GETZEKX356DMDGQO4HOA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.856523089 (151h11m38.552371842s) [noise=true , address=45.91.116.11:4117 , piece_id=QUGUYHEJWEEV733CBC3ONMV7XWDKVH2OFLQWXPQHPKZ7GTH4RLUA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.859490379 (151h11m38.555339122s) [noise=true , address=77.90.31.13:4628 , piece_id=QFTEHZ5FY7A5DPZZ44TRXGFRHPRNJZJHLQZDGRGOWYFEZYIZT2NQ, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.861350291 (151h11m38.557199034s) [noise=true , address=145.79.227.31:4797 , piece_id=2TFSZNURIAU573QUA5DUS7AJZNY2OXC6TJL3ST4VXIWY2WVB5YJA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.863368665 (151h11m38.559217418s) [noise=true , address=145.79.227.23:4547 , piece_id=K54H437KW5AAYUWOTMDBDP5GJ3ZABFKLAAOSS6WS5G5CROYEHODQ, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.864848794 (151h11m38.560697547s) [noise=true , address=145.79.227.24:4581 , piece_id=TSCVCH2KJEVMD4AVOEIHGTMRCRZ6NZMZA2CTQ6FUO2EIL3JAVYEA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.865644930 (151h11m38.561493683s) [noise=true , address=145.79.227.8:4330 , piece_id=4A47HIUX2MTJ2673XZZCMMF4T235TS4JWMNYJEVHHCKAHQX6APMA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:08.560504137 (151h11m40.25635289s) [upload=133242]: Next returned nil segment Nov 5 18:52:08.560522693 (151h11m40.256371446s) [upload=133242]: Waiting for error group managing segments... Nov 5 18:52:08.850333179 (151h11m40.546181932s) [noise=true , address=145.79.227.17:4972 , piece_id=H7UHQV7RDLCZUHY2XMXUAD2PX4ZBLWKTM47N6PH7UOGBRXPMWVOA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.853517822 (151h11m40.549366575s) [noise=true , address=145.79.227.22:4509 , piece_id=7RA5NJ72PI3DT4IC2ECVLZ3A2HXXL546ELVDXHZW4OXNN4D36OEA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.855547508 (151h11m40.551396261s) [noise=true , address=145.79.227.9:4366 , piece_id=7ZRECKBQC4S6QLERBZY442WYS4H35XWOZYSGGPHSU7NQ6WL4UNVA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.856103106 (151h11m40.551951859s) [noise=true , address=145.79.227.25:4618 , piece_id=XRUPY5BUPG7BXN2MI6MGYO4N3GW6TZDJE46MZFNRELWANHHG3N5A, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.860417321 (151h11m40.556266064s) [noise=true , address=145.79.227.8:4330 , piece_id=4A47HIUX2MTJ2673XZZCMMF4T235TS4JWMNYJEVHHCKAHQX6APMA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.865602796 (151h11m40.561451549s) [noise=true , address=145.79.227.23:4547 , piece_id=K54H437KW5AAYUWOTMDBDP5GJ3ZABFKLAAOSS6WS5G5CROYEHODQ, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.879393757 (151h11m40.57524251s) [noise=true , address=145.79.227.31:4797 , piece_id=2TFSZNURIAU573QUA5DUS7AJZNY2OXC6TJL3ST4VXIWY2WVB5YJA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.905541449 (151h11m40.601390202s) [noise=true , address=145.79.227.26:4654 , piece_id=E5J22KNPQSETSY443QQZXFFKUUJFHWFV22SQAVZ4BFQ4M6YWCNQQ, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.929018836 (151h11m40.624867589s) [noise=true , address=145.79.227.24:4581 , piece_id=TSCVCH2KJEVMD4AVOEIHGTMRCRZ6NZMZA2CTQ6FUO2EIL3JAVYEA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:35.899554917 (151h12m7.59540368s) [noise=true , address=77.90.31.17:4771 , piece_id=KVFE7VYW3RZJ25BWSH4BIRRQCMWDYOIK7EJ62XVREGN26YN7XYMQ, seg_pos={0 0}, upload=133242]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: write tcp 188.166.225.170:60676->77.90.31.17:4771: use of closed network connection; write tcp 188.166.225.170:60676->77.90.31.17:4771: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60676->77.90.31.17:4771: use of closed network connection Nov 5 18:52:35.899727506 (151h12m7.595576259s) [noise=true , address=77.90.31.13:4628 , piece_id=QFTEHZ5FY7A5DPZZ44TRXGFRHPRNJZJHLQZDGRGOWYFEZYIZT2NQ, seg_pos={0 0}, upload=133242]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: write tcp 188.166.225.170:42026->77.90.31.13:4628: use of closed network connection; write tcp 188.166.225.170:42026->77.90.31.13:4628: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42026->77.90.31.13:4628: use of closed network connection Nov 5 18:52:35.899860439 (151h12m7.595709192s) [noise=true , address=45.91.116.7:4010 , piece_id=EBVGGNQF3N4MTUIATJUWK4QZN7SEOWI6SIAW6N7JZOB5G6H33NEQ, seg_pos={0 0}, upload=133242]: Done uploading piece. err: ecclient: upload failed (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, address:45.91.116.7:4010): protocol: write tcp 188.166.225.170:40036->45.91.116.7:4010: use of closed network connection; write tcp 188.166.225.170:40036->45.91.116.7:4010: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40036->45.91.116.7:4010: use of closed network connection Nov 5 18:52:35.899923829 (151h12m7.595772572s) [noise=true , address=77.90.31.18:4805 , piece_id=U2U34KK4AJK4A5HGNTHEYCIARLAR4X5NLOVCF63TN3PJUIRMMIHA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: write tcp 188.166.225.170:60758->77.90.31.18:4805: use of closed network connection; write tcp 188.166.225.170:60758->77.90.31.18:4805: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60758->77.90.31.18:4805: use of closed network connection Nov 5 18:52:35.900052084 (151h12m7.595900837s) [noise=true , address=77.90.31.16:4734 , piece_id=HFWVFM46PZFLZT6YNZVIA6HKLVN2PTV2GETZEKX356DMDGQO4HOA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: write tcp 188.166.225.170:50728->77.90.31.16:4734: use of closed network connection; write tcp 188.166.225.170:50728->77.90.31.16:4734: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50728->77.90.31.16:4734: use of closed network connection Nov 5 18:52:35.900231175 (151h12m7.596079938s) [noise=true , address=45.91.116.11:4117 , piece_id=QUGUYHEJWEEV733CBC3ONMV7XWDKVH2OFLQWXPQHPKZ7GTH4RLUA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: ecclient: upload failed (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, address:45.91.116.11:4117): protocol: write tcp 188.166.225.170:41292->45.91.116.11:4117: use of closed network connection; write tcp 188.166.225.170:41292->45.91.116.11:4117: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41292->45.91.116.11:4117: use of closed network connection Nov 5 18:52:35.900342116 (151h12m7.596190859s) [noise=true , address=77.90.31.15:4698 , piece_id=7VACL26N7ZCBXIA2I3XL5Q3UJ2BAHDIPXX4VZCQX3S63ABNW625Q, seg_pos={0 0}, upload=133242]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: write tcp 188.166.225.170:41118->77.90.31.15:4698: use of closed network connection; write tcp 188.166.225.170:41118->77.90.31.15:4698: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41118->77.90.31.15:4698: use of closed network connection Nov 5 18:52:35.900382814 (151h12m7.596231567s) [seg_pos={0 0}, upload=133242]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Nov 5 18:52:36.162442033 (151h12m7.858290786s) [upload=133242]: Done uploading segments... Nov 5 18:52:36.162481568 (151h12m7.858330321s) [upload=133242]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 5.962703286s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [........................X..X................................] duration: 5.687878411s trace: https://jaeger.dev.storj.io/trace/3e2ae8884260b157 FAIL (2025-11-05 19:15:16 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.687909238s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [...........................XX...X...........................] duration: 3.46400482s trace: https://jaeger.dev.storj.io/trace/204ab2570dc82d9c FAIL (2025-11-05 19:14:59 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.464130188s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..........................X...X.............................] duration: 6.177226451s trace: https://jaeger.dev.storj.io/trace/74cf1e30f6d49950 FAIL (2025-11-05 19:14:45 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.177260313s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [........................X...X...............................] duration: 6.009333022s trace: https://jaeger.dev.storj.io/trace/2e576aae48e8f03e FAIL (2025-11-05 19:14:34 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== 5 18:52:06.850167887 (151h11m38.54601663s) [noise=true , address=145.79.227.25:4618 , piece_id=XRUPY5BUPG7BXN2MI6MGYO4N3GW6TZDJE46MZFNRELWANHHG3N5A, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.850747301 (151h11m38.546596054s) [noise=true , address=77.90.31.15:4698 , piece_id=7VACL26N7ZCBXIA2I3XL5Q3UJ2BAHDIPXX4VZCQX3S63ABNW625Q, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.851222605 (151h11m38.547071358s) [noise=true , address=145.79.227.17:4972 , piece_id=H7UHQV7RDLCZUHY2XMXUAD2PX4ZBLWKTM47N6PH7UOGBRXPMWVOA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.851558575 (151h11m38.547407319s) [noise=true , address=145.79.227.22:4509 , piece_id=7RA5NJ72PI3DT4IC2ECVLZ3A2HXXL546ELVDXHZW4OXNN4D36OEA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.851909263 (151h11m38.547758006s) [noise=true , address=145.79.227.26:4654 , piece_id=E5J22KNPQSETSY443QQZXFFKUUJFHWFV22SQAVZ4BFQ4M6YWCNQQ, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.852327540 (151h11m38.548176293s) [noise=true , address=45.91.116.7:4010 , piece_id=EBVGGNQF3N4MTUIATJUWK4QZN7SEOWI6SIAW6N7JZOB5G6H33NEQ, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.852728624 (151h11m38.548577377s) [noise=true , address=145.79.227.9:4366 , piece_id=7ZRECKBQC4S6QLERBZY442WYS4H35XWOZYSGGPHSU7NQ6WL4UNVA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.853978774 (151h11m38.549827528s) [noise=true , address=77.90.31.18:4805 , piece_id=U2U34KK4AJK4A5HGNTHEYCIARLAR4X5NLOVCF63TN3PJUIRMMIHA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.854447296 (151h11m38.550296059s) [noise=true , address=77.90.31.16:4734 , piece_id=HFWVFM46PZFLZT6YNZVIA6HKLVN2PTV2GETZEKX356DMDGQO4HOA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.856523089 (151h11m38.552371842s) [noise=true , address=45.91.116.11:4117 , piece_id=QUGUYHEJWEEV733CBC3ONMV7XWDKVH2OFLQWXPQHPKZ7GTH4RLUA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.859490379 (151h11m38.555339122s) [noise=true , address=77.90.31.13:4628 , piece_id=QFTEHZ5FY7A5DPZZ44TRXGFRHPRNJZJHLQZDGRGOWYFEZYIZT2NQ, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.861350291 (151h11m38.557199034s) [noise=true , address=145.79.227.31:4797 , piece_id=2TFSZNURIAU573QUA5DUS7AJZNY2OXC6TJL3ST4VXIWY2WVB5YJA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.863368665 (151h11m38.559217418s) [noise=true , address=145.79.227.23:4547 , piece_id=K54H437KW5AAYUWOTMDBDP5GJ3ZABFKLAAOSS6WS5G5CROYEHODQ, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.864848794 (151h11m38.560697547s) [noise=true , address=145.79.227.24:4581 , piece_id=TSCVCH2KJEVMD4AVOEIHGTMRCRZ6NZMZA2CTQ6FUO2EIL3JAVYEA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.865644930 (151h11m38.561493683s) [noise=true , address=145.79.227.8:4330 , piece_id=4A47HIUX2MTJ2673XZZCMMF4T235TS4JWMNYJEVHHCKAHQX6APMA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:08.560504137 (151h11m40.25635289s) [upload=133242]: Next returned nil segment Nov 5 18:52:08.560522693 (151h11m40.256371446s) [upload=133242]: Waiting for error group managing segments... Nov 5 18:52:08.850333179 (151h11m40.546181932s) [noise=true , address=145.79.227.17:4972 , piece_id=H7UHQV7RDLCZUHY2XMXUAD2PX4ZBLWKTM47N6PH7UOGBRXPMWVOA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.853517822 (151h11m40.549366575s) [noise=true , address=145.79.227.22:4509 , piece_id=7RA5NJ72PI3DT4IC2ECVLZ3A2HXXL546ELVDXHZW4OXNN4D36OEA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.855547508 (151h11m40.551396261s) [noise=true , address=145.79.227.9:4366 , piece_id=7ZRECKBQC4S6QLERBZY442WYS4H35XWOZYSGGPHSU7NQ6WL4UNVA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.856103106 (151h11m40.551951859s) [noise=true , address=145.79.227.25:4618 , piece_id=XRUPY5BUPG7BXN2MI6MGYO4N3GW6TZDJE46MZFNRELWANHHG3N5A, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.860417321 (151h11m40.556266064s) [noise=true , address=145.79.227.8:4330 , piece_id=4A47HIUX2MTJ2673XZZCMMF4T235TS4JWMNYJEVHHCKAHQX6APMA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.865602796 (151h11m40.561451549s) [noise=true , address=145.79.227.23:4547 , piece_id=K54H437KW5AAYUWOTMDBDP5GJ3ZABFKLAAOSS6WS5G5CROYEHODQ, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.879393757 (151h11m40.57524251s) [noise=true , address=145.79.227.31:4797 , piece_id=2TFSZNURIAU573QUA5DUS7AJZNY2OXC6TJL3ST4VXIWY2WVB5YJA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.905541449 (151h11m40.601390202s) [noise=true , address=145.79.227.26:4654 , piece_id=E5J22KNPQSETSY443QQZXFFKUUJFHWFV22SQAVZ4BFQ4M6YWCNQQ, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.929018836 (151h11m40.624867589s) [noise=true , address=145.79.227.24:4581 , piece_id=TSCVCH2KJEVMD4AVOEIHGTMRCRZ6NZMZA2CTQ6FUO2EIL3JAVYEA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:35.899554917 (151h12m7.59540368s) [noise=true , address=77.90.31.17:4771 , piece_id=KVFE7VYW3RZJ25BWSH4BIRRQCMWDYOIK7EJ62XVREGN26YN7XYMQ, seg_pos={0 0}, upload=133242]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: write tcp 188.166.225.170:60676->77.90.31.17:4771: use of closed network connection; write tcp 188.166.225.170:60676->77.90.31.17:4771: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60676->77.90.31.17:4771: use of closed network connection Nov 5 18:52:35.899727506 (151h12m7.595576259s) [noise=true , address=77.90.31.13:4628 , piece_id=QFTEHZ5FY7A5DPZZ44TRXGFRHPRNJZJHLQZDGRGOWYFEZYIZT2NQ, seg_pos={0 0}, upload=133242]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: write tcp 188.166.225.170:42026->77.90.31.13:4628: use of closed network connection; write tcp 188.166.225.170:42026->77.90.31.13:4628: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42026->77.90.31.13:4628: use of closed network connection Nov 5 18:52:35.899860439 (151h12m7.595709192s) [noise=true , address=45.91.116.7:4010 , piece_id=EBVGGNQF3N4MTUIATJUWK4QZN7SEOWI6SIAW6N7JZOB5G6H33NEQ, seg_pos={0 0}, upload=133242]: Done uploading piece. err: ecclient: upload failed (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, address:45.91.116.7:4010): protocol: write tcp 188.166.225.170:40036->45.91.116.7:4010: use of closed network connection; write tcp 188.166.225.170:40036->45.91.116.7:4010: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40036->45.91.116.7:4010: use of closed network connection Nov 5 18:52:35.899923829 (151h12m7.595772572s) [noise=true , address=77.90.31.18:4805 , piece_id=U2U34KK4AJK4A5HGNTHEYCIARLAR4X5NLOVCF63TN3PJUIRMMIHA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: write tcp 188.166.225.170:60758->77.90.31.18:4805: use of closed network connection; write tcp 188.166.225.170:60758->77.90.31.18:4805: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60758->77.90.31.18:4805: use of closed network connection Nov 5 18:52:35.900052084 (151h12m7.595900837s) [noise=true , address=77.90.31.16:4734 , piece_id=HFWVFM46PZFLZT6YNZVIA6HKLVN2PTV2GETZEKX356DMDGQO4HOA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: write tcp 188.166.225.170:50728->77.90.31.16:4734: use of closed network connection; write tcp 188.166.225.170:50728->77.90.31.16:4734: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50728->77.90.31.16:4734: use of closed network connection Nov 5 18:52:35.900231175 (151h12m7.596079938s) [noise=true , address=45.91.116.11:4117 , piece_id=QUGUYHEJWEEV733CBC3ONMV7XWDKVH2OFLQWXPQHPKZ7GTH4RLUA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: ecclient: upload failed (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, address:45.91.116.11:4117): protocol: write tcp 188.166.225.170:41292->45.91.116.11:4117: use of closed network connection; write tcp 188.166.225.170:41292->45.91.116.11:4117: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41292->45.91.116.11:4117: use of closed network connection Nov 5 18:52:35.900342116 (151h12m7.596190859s) [noise=true , address=77.90.31.15:4698 , piece_id=7VACL26N7ZCBXIA2I3XL5Q3UJ2BAHDIPXX4VZCQX3S63ABNW625Q, seg_pos={0 0}, upload=133242]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: write tcp 188.166.225.170:41118->77.90.31.15:4698: use of closed network connection; write tcp 188.166.225.170:41118->77.90.31.15:4698: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41118->77.90.31.15:4698: use of closed network connection Nov 5 18:52:35.900382814 (151h12m7.596231567s) [seg_pos={0 0}, upload=133242]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Nov 5 18:52:36.162442033 (151h12m7.858290786s) [upload=133242]: Done uploading segments... Nov 5 18:52:36.162481568 (151h12m7.858330321s) [upload=133242]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 6.009363691s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [............................XX...X..........................] duration: 3.588138817s trace: https://jaeger.dev.storj.io/trace/204ab2570dc82d9c FAIL (2025-11-05 19:14:21 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.58816627s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.........................X..X...............................] duration: 5.810696602s trace: https://jaeger.dev.storj.io/trace/3e2ae8884260b157 FAIL (2025-11-05 19:14:05 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.810725708s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.........................X...X..............................] duration: 5.884952934s trace: https://jaeger.dev.storj.io/trace/2e576aae48e8f03e FAIL (2025-11-05 19:13:58 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== 5 18:52:06.850167887 (151h11m38.54601663s) [noise=true , address=145.79.227.25:4618 , piece_id=XRUPY5BUPG7BXN2MI6MGYO4N3GW6TZDJE46MZFNRELWANHHG3N5A, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.850747301 (151h11m38.546596054s) [noise=true , address=77.90.31.15:4698 , piece_id=7VACL26N7ZCBXIA2I3XL5Q3UJ2BAHDIPXX4VZCQX3S63ABNW625Q, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.851222605 (151h11m38.547071358s) [noise=true , address=145.79.227.17:4972 , piece_id=H7UHQV7RDLCZUHY2XMXUAD2PX4ZBLWKTM47N6PH7UOGBRXPMWVOA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.851558575 (151h11m38.547407319s) [noise=true , address=145.79.227.22:4509 , piece_id=7RA5NJ72PI3DT4IC2ECVLZ3A2HXXL546ELVDXHZW4OXNN4D36OEA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.851909263 (151h11m38.547758006s) [noise=true , address=145.79.227.26:4654 , piece_id=E5J22KNPQSETSY443QQZXFFKUUJFHWFV22SQAVZ4BFQ4M6YWCNQQ, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.852327540 (151h11m38.548176293s) [noise=true , address=45.91.116.7:4010 , piece_id=EBVGGNQF3N4MTUIATJUWK4QZN7SEOWI6SIAW6N7JZOB5G6H33NEQ, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.852728624 (151h11m38.548577377s) [noise=true , address=145.79.227.9:4366 , piece_id=7ZRECKBQC4S6QLERBZY442WYS4H35XWOZYSGGPHSU7NQ6WL4UNVA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.853978774 (151h11m38.549827528s) [noise=true , address=77.90.31.18:4805 , piece_id=U2U34KK4AJK4A5HGNTHEYCIARLAR4X5NLOVCF63TN3PJUIRMMIHA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.854447296 (151h11m38.550296059s) [noise=true , address=77.90.31.16:4734 , piece_id=HFWVFM46PZFLZT6YNZVIA6HKLVN2PTV2GETZEKX356DMDGQO4HOA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.856523089 (151h11m38.552371842s) [noise=true , address=45.91.116.11:4117 , piece_id=QUGUYHEJWEEV733CBC3ONMV7XWDKVH2OFLQWXPQHPKZ7GTH4RLUA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.859490379 (151h11m38.555339122s) [noise=true , address=77.90.31.13:4628 , piece_id=QFTEHZ5FY7A5DPZZ44TRXGFRHPRNJZJHLQZDGRGOWYFEZYIZT2NQ, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.861350291 (151h11m38.557199034s) [noise=true , address=145.79.227.31:4797 , piece_id=2TFSZNURIAU573QUA5DUS7AJZNY2OXC6TJL3ST4VXIWY2WVB5YJA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.863368665 (151h11m38.559217418s) [noise=true , address=145.79.227.23:4547 , piece_id=K54H437KW5AAYUWOTMDBDP5GJ3ZABFKLAAOSS6WS5G5CROYEHODQ, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.864848794 (151h11m38.560697547s) [noise=true , address=145.79.227.24:4581 , piece_id=TSCVCH2KJEVMD4AVOEIHGTMRCRZ6NZMZA2CTQ6FUO2EIL3JAVYEA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:06.865644930 (151h11m38.561493683s) [noise=true , address=145.79.227.8:4330 , piece_id=4A47HIUX2MTJ2673XZZCMMF4T235TS4JWMNYJEVHHCKAHQX6APMA, seg_pos={0 0}, upload=133242]: Uploading piece... Nov 5 18:52:08.560504137 (151h11m40.25635289s) [upload=133242]: Next returned nil segment Nov 5 18:52:08.560522693 (151h11m40.256371446s) [upload=133242]: Waiting for error group managing segments... Nov 5 18:52:08.850333179 (151h11m40.546181932s) [noise=true , address=145.79.227.17:4972 , piece_id=H7UHQV7RDLCZUHY2XMXUAD2PX4ZBLWKTM47N6PH7UOGBRXPMWVOA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.853517822 (151h11m40.549366575s) [noise=true , address=145.79.227.22:4509 , piece_id=7RA5NJ72PI3DT4IC2ECVLZ3A2HXXL546ELVDXHZW4OXNN4D36OEA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.855547508 (151h11m40.551396261s) [noise=true , address=145.79.227.9:4366 , piece_id=7ZRECKBQC4S6QLERBZY442WYS4H35XWOZYSGGPHSU7NQ6WL4UNVA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.856103106 (151h11m40.551951859s) [noise=true , address=145.79.227.25:4618 , piece_id=XRUPY5BUPG7BXN2MI6MGYO4N3GW6TZDJE46MZFNRELWANHHG3N5A, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.860417321 (151h11m40.556266064s) [noise=true , address=145.79.227.8:4330 , piece_id=4A47HIUX2MTJ2673XZZCMMF4T235TS4JWMNYJEVHHCKAHQX6APMA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.865602796 (151h11m40.561451549s) [noise=true , address=145.79.227.23:4547 , piece_id=K54H437KW5AAYUWOTMDBDP5GJ3ZABFKLAAOSS6WS5G5CROYEHODQ, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.879393757 (151h11m40.57524251s) [noise=true , address=145.79.227.31:4797 , piece_id=2TFSZNURIAU573QUA5DUS7AJZNY2OXC6TJL3ST4VXIWY2WVB5YJA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.905541449 (151h11m40.601390202s) [noise=true , address=145.79.227.26:4654 , piece_id=E5J22KNPQSETSY443QQZXFFKUUJFHWFV22SQAVZ4BFQ4M6YWCNQQ, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:08.929018836 (151h11m40.624867589s) [noise=true , address=145.79.227.24:4581 , piece_id=TSCVCH2KJEVMD4AVOEIHGTMRCRZ6NZMZA2CTQ6FUO2EIL3JAVYEA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: <nil> Nov 5 18:52:35.899554917 (151h12m7.59540368s) [noise=true , address=77.90.31.17:4771 , piece_id=KVFE7VYW3RZJ25BWSH4BIRRQCMWDYOIK7EJ62XVREGN26YN7XYMQ, seg_pos={0 0}, upload=133242]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: write tcp 188.166.225.170:60676->77.90.31.17:4771: use of closed network connection; write tcp 188.166.225.170:60676->77.90.31.17:4771: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60676->77.90.31.17:4771: use of closed network connection Nov 5 18:52:35.899727506 (151h12m7.595576259s) [noise=true , address=77.90.31.13:4628 , piece_id=QFTEHZ5FY7A5DPZZ44TRXGFRHPRNJZJHLQZDGRGOWYFEZYIZT2NQ, seg_pos={0 0}, upload=133242]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: write tcp 188.166.225.170:42026->77.90.31.13:4628: use of closed network connection; write tcp 188.166.225.170:42026->77.90.31.13:4628: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42026->77.90.31.13:4628: use of closed network connection Nov 5 18:52:35.899860439 (151h12m7.595709192s) [noise=true , address=45.91.116.7:4010 , piece_id=EBVGGNQF3N4MTUIATJUWK4QZN7SEOWI6SIAW6N7JZOB5G6H33NEQ, seg_pos={0 0}, upload=133242]: Done uploading piece. err: ecclient: upload failed (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, address:45.91.116.7:4010): protocol: write tcp 188.166.225.170:40036->45.91.116.7:4010: use of closed network connection; write tcp 188.166.225.170:40036->45.91.116.7:4010: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40036->45.91.116.7:4010: use of closed network connection Nov 5 18:52:35.899923829 (151h12m7.595772572s) [noise=true , address=77.90.31.18:4805 , piece_id=U2U34KK4AJK4A5HGNTHEYCIARLAR4X5NLOVCF63TN3PJUIRMMIHA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: write tcp 188.166.225.170:60758->77.90.31.18:4805: use of closed network connection; write tcp 188.166.225.170:60758->77.90.31.18:4805: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60758->77.90.31.18:4805: use of closed network connection Nov 5 18:52:35.900052084 (151h12m7.595900837s) [noise=true , address=77.90.31.16:4734 , piece_id=HFWVFM46PZFLZT6YNZVIA6HKLVN2PTV2GETZEKX356DMDGQO4HOA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: write tcp 188.166.225.170:50728->77.90.31.16:4734: use of closed network connection; write tcp 188.166.225.170:50728->77.90.31.16:4734: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50728->77.90.31.16:4734: use of closed network connection Nov 5 18:52:35.900231175 (151h12m7.596079938s) [noise=true , address=45.91.116.11:4117 , piece_id=QUGUYHEJWEEV733CBC3ONMV7XWDKVH2OFLQWXPQHPKZ7GTH4RLUA, seg_pos={0 0}, upload=133242]: Done uploading piece. err: ecclient: upload failed (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, address:45.91.116.11:4117): protocol: write tcp 188.166.225.170:41292->45.91.116.11:4117: use of closed network connection; write tcp 188.166.225.170:41292->45.91.116.11:4117: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41292->45.91.116.11:4117: use of closed network connection Nov 5 18:52:35.900342116 (151h12m7.596190859s) [noise=true , address=77.90.31.15:4698 , piece_id=7VACL26N7ZCBXIA2I3XL5Q3UJ2BAHDIPXX4VZCQX3S63ABNW625Q, seg_pos={0 0}, upload=133242]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: write tcp 188.166.225.170:41118->77.90.31.15:4698: use of closed network connection; write tcp 188.166.225.170:41118->77.90.31.15:4698: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41118->77.90.31.15:4698: use of closed network connection Nov 5 18:52:35.900382814 (151h12m7.596231567s) [seg_pos={0 0}, upload=133242]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Nov 5 18:52:36.162442033 (151h12m7.858290786s) [upload=133242]: Done uploading segments... Nov 5 18:52:36.162481568 (151h12m7.858330321s) [upload=133242]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 5.884971359s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...........................X...X............................] duration: 6.458867435s trace: https://jaeger.dev.storj.io/trace/74cf1e30f6d49950 FAIL (2025-11-05 19:13:51 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.458888715s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [.............................XX...X.........................] duration: 3.549549314s trace: https://jaeger.dev.storj.io/trace/204ab2570dc82d9c FAIL (2025-11-05 19:13:50 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.549597725s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [..........................X..X..............................] duration: 6.02994512s trace: https://jaeger.dev.storj.io/trace/3e2ae8884260b157 FAIL (2025-11-05 19:13:05 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.029975729s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)