Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 5.840126649s FAIL (2025-10-27 15:43:17 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, Piece ID: 76MPB5O5XOR54TOXYCUNK2RBQMWYTWADCBAZV2KQYJ5K7XXDVFMQ) context deadline exceeded; piecestore: (Node ID: 12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, Piece ID: GR3XECVFMZ5E3HDMKK2HLKJREMIOZGWIUI7J3NSK7QGB523G7YJQ) context deadline exceeded; piecestore: (Node ID: 12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, Piece ID: FHAUWKUQAHGYR6ZJHKBCXT3ZPBG42MMJSADUGU7CIFUO3JPPNKKA) context deadline exceeded; piecestore: (Node ID: 1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV, Piece ID: 2EGSCSNCQJFXYT54PWEM33SP3RUYICBKKE7IDRQNMZ2ONPM6I5UA) context deadline exceeded ======LOGS====== ]: Done starting upload Oct 27 15:28:22.900340512 (45m40.249647808s) [upload=668]: Uploading segments... Oct 27 15:28:22.900564057 (45m40.249871343s) [upload=668]: Got next segment. Inline: false Oct 27 15:28:22.900613051 (45m40.249920337s) [upload=668]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 27 15:28:22.900620896 (45m40.249928182s) [upload=668]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 27 15:28:23.560720563 (45m40.91002784s) [seg_pos={0 0}, upload=668]: Begin upload segment... Oct 27 15:28:23.560802119 (45m40.910109405s) [seg_pos={0 0}, upload=668]: Done begin upload segment. Oct 27 15:28:23.560839660 (45m40.910146947s) [noise=true , address=145.79.227.20:4437 , piece_id=2EGSCSNCQJFXYT54PWEM33SP3RUYICBKKE7IDRQNMZ2ONPM6I5UA, seg_pos={0 0}, upload=668]: Uploading piece... Oct 27 15:28:23.560885057 (45m40.910192353s) [noise=true , address=145.79.227.19:4079 , piece_id=QXNYHJS2SBTG2QRH7AKW7NZ6SJG7MFO5DHXCMBZHGT6PJTOZ526Q, seg_pos={0 0}, upload=668]: Uploading piece... Oct 27 15:28:23.561607883 (45m40.910915169s) [noise=true , address=145.79.227.26:4653 , piece_id=MXREE4NCFOGYEHVQNFHZYD5DLD4PHBJS64JYSLTXXJZG7AC3LH6Q, seg_pos={0 0}, upload=668]: Uploading piece... Oct 27 15:28:23.561670783 (45m40.910978069s) [noise=true , address=77.90.31.10:4518 , piece_id=76MPB5O5XOR54TOXYCUNK2RBQMWYTWADCBAZV2KQYJ5K7XXDVFMQ, seg_pos={0 0}, upload=668]: Uploading piece... Oct 27 15:28:23.561966716 (45m40.911274002s) [noise=true , address=145.79.227.10:4401 , piece_id=SZNSLFKN44YVSUHRXZTF6BBVPYQODYVEBWQF3JHP6ZDA2A723WAA, seg_pos={0 0}, upload=668]: Uploading piece... Oct 27 15:28:23.562189701 (45m40.911496987s) [noise=true , address=145.79.227.4:4186 , piece_id=6APC73J3JHGVX3ZNCN3UL5DLHXRYORPQT6AYPFPX6ZQLQIRF5TNA, seg_pos={0 0}, upload=668]: Uploading piece... Oct 27 15:28:23.562277909 (45m40.911585195s) [noise=true , address=145.79.227.7:4294 , piece_id=QFLUZYLIGHO5E3ABEA5LFJPHAKDOF3J55Q4BRF5NXVKDNEHV5DBA, seg_pos={0 0}, upload=668]: Uploading piece... Oct 27 15:28:23.562572600 (45m40.911879887s) [noise=true , address=145.79.227.5:4222 , piece_id=B7W6PCNLLAPHXL7XOSJXEOVX6XZGNFDJWXR7MVYDGCONQCN55HYA, seg_pos={0 0}, upload=668]: Uploading piece... Oct 27 15:28:23.562586527 (45m40.911893813s) [noise=true , address=145.79.227.9:4366 , piece_id=5XLRE4OGUCQGZA5UXVNTNTFAMAGGQNHRR4MX23L3362K2DZLDREQ, seg_pos={0 0}, upload=668]: Uploading piece... Oct 27 15:28:23.562888732 (45m40.912196018s) [noise=true , address=145.79.227.21:4475 , piece_id=VDOFRCJKDQLXV4CBM4MA3UGK2PZMRJ3BW6H7PKJRZMZXW56Y5G4A, seg_pos={0 0}, upload=668]: Uploading piece... Oct 27 15:28:23.562953846 (45m40.912261123s) [noise=true , address=145.79.227.17:4971 , piece_id=NK55UPWQTKDWAAMPQTJPPOMZXH5VQGQ3QHJKRNMIRPFLFKQBZZIA, seg_pos={0 0}, upload=668]: Uploading piece... Oct 27 15:28:23.563182882 (45m40.912490158s) [noise=true , address=77.90.31.18:4807 , piece_id=OPDBUFUCTPZN2HH2SORHJL5L4EX2VWXATEFEIIEN5QTBQQFRGSZA, seg_pos={0 0}, upload=668]: Uploading piece... Oct 27 15:28:23.563335423 (45m40.912642709s) [noise=true , address=145.79.227.24:4582 , piece_id=K7COJQ2UG3CTGF4JVO27PTGWY4E3XGGRQWKC3LW24Y565B52YJCQ, seg_pos={0 0}, upload=668]: Uploading piece... Oct 27 15:28:23.563479036 (45m40.912786323s) [noise=true , address=77.90.31.13:4626 , piece_id=GR3XECVFMZ5E3HDMKK2HLKJREMIOZGWIUI7J3NSK7QGB523G7YJQ, seg_pos={0 0}, upload=668]: Uploading piece... Oct 27 15:28:23.563711468 (45m40.913018754s) [noise=true , address=45.91.116.5:4298 , piece_id=ATXRIPPIX6BAHUXCENJVDJ25K2Z6UH3QYL4MTBWPZKP7LFGHBX6A, seg_pos={0 0}, upload=668]: Uploading piece... Oct 27 15:28:23.564137260 (45m40.913444546s) [noise=true , address=145.79.227.5:4223 , piece_id=FHAUWKUQAHGYR6ZJHKBCXT3ZPBG42MMJSADUGU7CIFUO3JPPNKKA, seg_pos={0 0}, upload=668]: Uploading piece... Oct 27 15:28:27.241369967 (45m44.590677243s) [upload=668]: Next returned nil segment Oct 27 15:28:27.241385446 (45m44.590692732s) [upload=668]: Waiting for error group managing segments... Oct 27 15:28:29.744963349 (45m47.094270636s) [noise=true , address=145.79.227.5:4222 , piece_id=B7W6PCNLLAPHXL7XOSJXEOVX6XZGNFDJWXR7MVYDGCONQCN55HYA, seg_pos={0 0}, upload=668]: Done uploading piece. err: <nil> Oct 27 15:28:29.821202504 (45m47.17050978s) [noise=true , address=145.79.227.21:4475 , piece_id=VDOFRCJKDQLXV4CBM4MA3UGK2PZMRJ3BW6H7PKJRZMZXW56Y5G4A, seg_pos={0 0}, upload=668]: Done uploading piece. err: <nil> Oct 27 15:28:32.126653324 (45m49.47596061s) [noise=true , address=77.90.31.18:4807 , piece_id=OPDBUFUCTPZN2HH2SORHJL5L4EX2VWXATEFEIIEN5QTBQQFRGSZA, seg_pos={0 0}, upload=668]: Done uploading piece. err: <nil> Oct 27 15:28:33.087272849 (45m50.436580125s) [noise=true , address=45.91.116.5:4298 , piece_id=ATXRIPPIX6BAHUXCENJVDJ25K2Z6UH3QYL4MTBWPZKP7LFGHBX6A, seg_pos={0 0}, upload=668]: Done uploading piece. err: <nil> Oct 27 15:28:38.581899084 (45m55.93120637s) [noise=true , address=145.79.227.26:4653 , piece_id=MXREE4NCFOGYEHVQNFHZYD5DLD4PHBJS64JYSLTXXJZG7AC3LH6Q, seg_pos={0 0}, upload=668]: Done uploading piece. err: <nil> Oct 27 15:28:39.053322174 (45m56.40262946s) [noise=true , address=145.79.227.24:4582 , piece_id=K7COJQ2UG3CTGF4JVO27PTGWY4E3XGGRQWKC3LW24Y565B52YJCQ, seg_pos={0 0}, upload=668]: Done uploading piece. err: <nil> Oct 27 15:28:39.717124420 (45m57.066431696s) [noise=true , address=145.79.227.20:4437 , piece_id=2EGSCSNCQJFXYT54PWEM33SP3RUYICBKKE7IDRQNMZ2ONPM6I5UA, seg_pos={0 0}, upload=668]: Done uploading piece. err: <nil> Oct 27 15:28:41.618415745 (45m58.967723031s) [noise=true , address=77.90.31.13:4626 , piece_id=GR3XECVFMZ5E3HDMKK2HLKJREMIOZGWIUI7J3NSK7QGB523G7YJQ, seg_pos={0 0}, upload=668]: Done uploading piece. err: <nil> Oct 27 15:28:42.068020890 (45m59.417328166s) [noise=true , address=145.79.227.5:4223 , piece_id=FHAUWKUQAHGYR6ZJHKBCXT3ZPBG42MMJSADUGU7CIFUO3JPPNKKA, seg_pos={0 0}, upload=668]: Done uploading piece. err: <nil> Oct 27 15:28:42.150893235 (45m59.500200521s) [noise=true , address=77.90.31.10:4518 , piece_id=76MPB5O5XOR54TOXYCUNK2RBQMWYTWADCBAZV2KQYJ5K7XXDVFMQ, seg_pos={0 0}, upload=668]: Done uploading piece. err: <nil> Oct 27 15:28:43.703285619 (46m1.052592905s) [noise=true , address=145.79.227.9:4366 , piece_id=5XLRE4OGUCQGZA5UXVNTNTFAMAGGQNHRR4MX23L3362K2DZLDREQ, seg_pos={0 0}, upload=668]: Done uploading piece. err: <nil> Oct 27 15:28:49.699904003 (46m7.049211289s) [noise=true , address=145.79.227.4:4186 , piece_id=6APC73J3JHGVX3ZNCN3UL5DLHXRYORPQT6AYPFPX6ZQLQIRF5TNA, seg_pos={0 0}, upload=668]: Done uploading piece. err: <nil> Oct 27 15:28:49.842045738 (46m7.191353014s) [noise=true , address=145.79.227.7:4294 , piece_id=QFLUZYLIGHO5E3ABEA5LFJPHAKDOF3J55Q4BRF5NXVKDNEHV5DBA, seg_pos={0 0}, upload=668]: Done uploading piece. err: <nil> Oct 27 15:28:49.842081656 (46m7.191388932s) [seg_pos={0 0}, upload=668]: Segment reached optimal threshold of 13 pieces. Oct 27 15:28:49.842296495 (46m7.191603781s) [noise=true , address=145.79.227.17:4971 , piece_id=NK55UPWQTKDWAAMPQTJPPOMZXH5VQGQ3QHJKRNMIRPFLFKQBZZIA, seg_pos={0 0}, upload=668]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo): protocol: write tcp 188.166.225.170:55424->145.79.227.17:4971: use of closed network connection; write tcp 188.166.225.170:55424->145.79.227.17:4971: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55424->145.79.227.17:4971: use of closed network connection Oct 27 15:28:49.842306875 (46m7.191614161s) [noise=true , address=145.79.227.17:4971 , piece_id=NK55UPWQTKDWAAMPQTJPPOMZXH5VQGQ3QHJKRNMIRPFLFKQBZZIA, seg_pos={0 0}, upload=668]: Long tail cancellation due to optimal threshold Oct 27 15:28:49.842335229 (46m7.191642535s) [noise=true , address=145.79.227.10:4401 , piece_id=SZNSLFKN44YVSUHRXZTF6BBVPYQODYVEBWQF3JHP6ZDA2A723WAA, seg_pos={0 0}, upload=668]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1): protocol: write tcp 188.166.225.170:50900->145.79.227.10:4401: use of closed network connection; write tcp 188.166.225.170:50900->145.79.227.10:4401: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50900->145.79.227.10:4401: use of closed network connection Oct 27 15:28:49.842355578 (46m7.191662874s) [noise=true , address=145.79.227.10:4401 , piece_id=SZNSLFKN44YVSUHRXZTF6BBVPYQODYVEBWQF3JHP6ZDA2A723WAA, seg_pos={0 0}, upload=668]: Long tail cancellation due to optimal threshold Oct 27 15:28:49.842403068 (46m7.191710354s) [noise=true , address=145.79.227.19:4079 , piece_id=QXNYHJS2SBTG2QRH7AKW7NZ6SJG7MFO5DHXCMBZHGT6PJTOZ526Q, seg_pos={0 0}, upload=668]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud): protocol: write tcp 188.166.225.170:37220->145.79.227.19:4079: use of closed network connection; write tcp 188.166.225.170:37220->145.79.227.19:4079: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37220->145.79.227.19:4079: use of closed network connection Oct 27 15:28:49.842411143 (46m7.191718419s) [noise=true , address=145.79.227.19:4079 , piece_id=QXNYHJS2SBTG2QRH7AKW7NZ6SJG7MFO5DHXCMBZHGT6PJTOZ526Q, seg_pos={0 0}, upload=668]: Long tail cancellation due to optimal threshold Oct 27 15:28:49.842428026 (46m7.191735312s) [seg_pos={0 0}, upload=668]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 27 15:28:49.842507998 (46m7.191815284s) [upload=668]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 27 15:28:49.842513478 (46m7.191820764s) [upload=668]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 27 15:28:50.170126807 (46m7.519434093s) [upload=668]: Done uploading segments... Oct 27 15:28:50.170148138 (46m7.519455424s) [upload=668]: Upload finished. err: <nil> Duration: 5.840156204s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 3/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=3) [........X..............XX................X..................] duration: 3.409739735s FAIL (2025-10-27 15:43:11 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.40976375s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 5.71931611s FAIL (2025-10-27 15:43:10 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.719338592s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 5.824289219s FAIL (2025-10-27 15:42:47 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, Piece ID: 76MPB5O5XOR54TOXYCUNK2RBQMWYTWADCBAZV2KQYJ5K7XXDVFMQ) context deadline exceeded; piecestore: (Node ID: 12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, Piece ID: GR3XECVFMZ5E3HDMKK2HLKJREMIOZGWIUI7J3NSK7QGB523G7YJQ) context deadline exceeded; piecestore: (Node ID: 12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, Piece ID: FHAUWKUQAHGYR6ZJHKBCXT3ZPBG42MMJSADUGU7CIFUO3JPPNKKA) context deadline exceeded; piecestore: (Node ID: 1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV, Piece ID: 2EGSCSNCQJFXYT54PWEM33SP3RUYICBKKE7IDRQNMZ2ONPM6I5UA) context deadline exceeded ======LOGS====== ]: Done starting upload Oct 27 15:28:22.900340512 (45m40.249647808s) [upload=668]: Uploading segments... Oct 27 15:28:22.900564057 (45m40.249871343s) [upload=668]: Got next segment. Inline: false Oct 27 15:28:22.900613051 (45m40.249920337s) [upload=668]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 27 15:28:22.900620896 (45m40.249928182s) [upload=668]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 27 15:28:23.560720563 (45m40.91002784s) [seg_pos={0 0}, upload=668]: Begin upload segment... Oct 27 15:28:23.560802119 (45m40.910109405s) [seg_pos={0 0}, upload=668]: Done begin upload segment. Oct 27 15:28:23.560839660 (45m40.910146947s) [noise=true , address=145.79.227.20:4437 , piece_id=2EGSCSNCQJFXYT54PWEM33SP3RUYICBKKE7IDRQNMZ2ONPM6I5UA, seg_pos={0 0}, upload=668]: Uploading piece... Oct 27 15:28:23.560885057 (45m40.910192353s) [noise=true , address=145.79.227.19:4079 , piece_id=QXNYHJS2SBTG2QRH7AKW7NZ6SJG7MFO5DHXCMBZHGT6PJTOZ526Q, seg_pos={0 0}, upload=668]: Uploading piece... Oct 27 15:28:23.561607883 (45m40.910915169s) [noise=true , address=145.79.227.26:4653 , piece_id=MXREE4NCFOGYEHVQNFHZYD5DLD4PHBJS64JYSLTXXJZG7AC3LH6Q, seg_pos={0 0}, upload=668]: Uploading piece... Oct 27 15:28:23.561670783 (45m40.910978069s) [noise=true , address=77.90.31.10:4518 , piece_id=76MPB5O5XOR54TOXYCUNK2RBQMWYTWADCBAZV2KQYJ5K7XXDVFMQ, seg_pos={0 0}, upload=668]: Uploading piece... Oct 27 15:28:23.561966716 (45m40.911274002s) [noise=true , address=145.79.227.10:4401 , piece_id=SZNSLFKN44YVSUHRXZTF6BBVPYQODYVEBWQF3JHP6ZDA2A723WAA, seg_pos={0 0}, upload=668]: Uploading piece... Oct 27 15:28:23.562189701 (45m40.911496987s) [noise=true , address=145.79.227.4:4186 , piece_id=6APC73J3JHGVX3ZNCN3UL5DLHXRYORPQT6AYPFPX6ZQLQIRF5TNA, seg_pos={0 0}, upload=668]: Uploading piece... Oct 27 15:28:23.562277909 (45m40.911585195s) [noise=true , address=145.79.227.7:4294 , piece_id=QFLUZYLIGHO5E3ABEA5LFJPHAKDOF3J55Q4BRF5NXVKDNEHV5DBA, seg_pos={0 0}, upload=668]: Uploading piece... Oct 27 15:28:23.562572600 (45m40.911879887s) [noise=true , address=145.79.227.5:4222 , piece_id=B7W6PCNLLAPHXL7XOSJXEOVX6XZGNFDJWXR7MVYDGCONQCN55HYA, seg_pos={0 0}, upload=668]: Uploading piece... Oct 27 15:28:23.562586527 (45m40.911893813s) [noise=true , address=145.79.227.9:4366 , piece_id=5XLRE4OGUCQGZA5UXVNTNTFAMAGGQNHRR4MX23L3362K2DZLDREQ, seg_pos={0 0}, upload=668]: Uploading piece... Oct 27 15:28:23.562888732 (45m40.912196018s) [noise=true , address=145.79.227.21:4475 , piece_id=VDOFRCJKDQLXV4CBM4MA3UGK2PZMRJ3BW6H7PKJRZMZXW56Y5G4A, seg_pos={0 0}, upload=668]: Uploading piece... Oct 27 15:28:23.562953846 (45m40.912261123s) [noise=true , address=145.79.227.17:4971 , piece_id=NK55UPWQTKDWAAMPQTJPPOMZXH5VQGQ3QHJKRNMIRPFLFKQBZZIA, seg_pos={0 0}, upload=668]: Uploading piece... Oct 27 15:28:23.563182882 (45m40.912490158s) [noise=true , address=77.90.31.18:4807 , piece_id=OPDBUFUCTPZN2HH2SORHJL5L4EX2VWXATEFEIIEN5QTBQQFRGSZA, seg_pos={0 0}, upload=668]: Uploading piece... Oct 27 15:28:23.563335423 (45m40.912642709s) [noise=true , address=145.79.227.24:4582 , piece_id=K7COJQ2UG3CTGF4JVO27PTGWY4E3XGGRQWKC3LW24Y565B52YJCQ, seg_pos={0 0}, upload=668]: Uploading piece... Oct 27 15:28:23.563479036 (45m40.912786323s) [noise=true , address=77.90.31.13:4626 , piece_id=GR3XECVFMZ5E3HDMKK2HLKJREMIOZGWIUI7J3NSK7QGB523G7YJQ, seg_pos={0 0}, upload=668]: Uploading piece... Oct 27 15:28:23.563711468 (45m40.913018754s) [noise=true , address=45.91.116.5:4298 , piece_id=ATXRIPPIX6BAHUXCENJVDJ25K2Z6UH3QYL4MTBWPZKP7LFGHBX6A, seg_pos={0 0}, upload=668]: Uploading piece... Oct 27 15:28:23.564137260 (45m40.913444546s) [noise=true , address=145.79.227.5:4223 , piece_id=FHAUWKUQAHGYR6ZJHKBCXT3ZPBG42MMJSADUGU7CIFUO3JPPNKKA, seg_pos={0 0}, upload=668]: Uploading piece... Oct 27 15:28:27.241369967 (45m44.590677243s) [upload=668]: Next returned nil segment Oct 27 15:28:27.241385446 (45m44.590692732s) [upload=668]: Waiting for error group managing segments... Oct 27 15:28:29.744963349 (45m47.094270636s) [noise=true , address=145.79.227.5:4222 , piece_id=B7W6PCNLLAPHXL7XOSJXEOVX6XZGNFDJWXR7MVYDGCONQCN55HYA, seg_pos={0 0}, upload=668]: Done uploading piece. err: <nil> Oct 27 15:28:29.821202504 (45m47.17050978s) [noise=true , address=145.79.227.21:4475 , piece_id=VDOFRCJKDQLXV4CBM4MA3UGK2PZMRJ3BW6H7PKJRZMZXW56Y5G4A, seg_pos={0 0}, upload=668]: Done uploading piece. err: <nil> Oct 27 15:28:32.126653324 (45m49.47596061s) [noise=true , address=77.90.31.18:4807 , piece_id=OPDBUFUCTPZN2HH2SORHJL5L4EX2VWXATEFEIIEN5QTBQQFRGSZA, seg_pos={0 0}, upload=668]: Done uploading piece. err: <nil> Oct 27 15:28:33.087272849 (45m50.436580125s) [noise=true , address=45.91.116.5:4298 , piece_id=ATXRIPPIX6BAHUXCENJVDJ25K2Z6UH3QYL4MTBWPZKP7LFGHBX6A, seg_pos={0 0}, upload=668]: Done uploading piece. err: <nil> Oct 27 15:28:38.581899084 (45m55.93120637s) [noise=true , address=145.79.227.26:4653 , piece_id=MXREE4NCFOGYEHVQNFHZYD5DLD4PHBJS64JYSLTXXJZG7AC3LH6Q, seg_pos={0 0}, upload=668]: Done uploading piece. err: <nil> Oct 27 15:28:39.053322174 (45m56.40262946s) [noise=true , address=145.79.227.24:4582 , piece_id=K7COJQ2UG3CTGF4JVO27PTGWY4E3XGGRQWKC3LW24Y565B52YJCQ, seg_pos={0 0}, upload=668]: Done uploading piece. err: <nil> Oct 27 15:28:39.717124420 (45m57.066431696s) [noise=true , address=145.79.227.20:4437 , piece_id=2EGSCSNCQJFXYT54PWEM33SP3RUYICBKKE7IDRQNMZ2ONPM6I5UA, seg_pos={0 0}, upload=668]: Done uploading piece. err: <nil> Oct 27 15:28:41.618415745 (45m58.967723031s) [noise=true , address=77.90.31.13:4626 , piece_id=GR3XECVFMZ5E3HDMKK2HLKJREMIOZGWIUI7J3NSK7QGB523G7YJQ, seg_pos={0 0}, upload=668]: Done uploading piece. err: <nil> Oct 27 15:28:42.068020890 (45m59.417328166s) [noise=true , address=145.79.227.5:4223 , piece_id=FHAUWKUQAHGYR6ZJHKBCXT3ZPBG42MMJSADUGU7CIFUO3JPPNKKA, seg_pos={0 0}, upload=668]: Done uploading piece. err: <nil> Oct 27 15:28:42.150893235 (45m59.500200521s) [noise=true , address=77.90.31.10:4518 , piece_id=76MPB5O5XOR54TOXYCUNK2RBQMWYTWADCBAZV2KQYJ5K7XXDVFMQ, seg_pos={0 0}, upload=668]: Done uploading piece. err: <nil> Oct 27 15:28:43.703285619 (46m1.052592905s) [noise=true , address=145.79.227.9:4366 , piece_id=5XLRE4OGUCQGZA5UXVNTNTFAMAGGQNHRR4MX23L3362K2DZLDREQ, seg_pos={0 0}, upload=668]: Done uploading piece. err: <nil> Oct 27 15:28:49.699904003 (46m7.049211289s) [noise=true , address=145.79.227.4:4186 , piece_id=6APC73J3JHGVX3ZNCN3UL5DLHXRYORPQT6AYPFPX6ZQLQIRF5TNA, seg_pos={0 0}, upload=668]: Done uploading piece. err: <nil> Oct 27 15:28:49.842045738 (46m7.191353014s) [noise=true , address=145.79.227.7:4294 , piece_id=QFLUZYLIGHO5E3ABEA5LFJPHAKDOF3J55Q4BRF5NXVKDNEHV5DBA, seg_pos={0 0}, upload=668]: Done uploading piece. err: <nil> Oct 27 15:28:49.842081656 (46m7.191388932s) [seg_pos={0 0}, upload=668]: Segment reached optimal threshold of 13 pieces. Oct 27 15:28:49.842296495 (46m7.191603781s) [noise=true , address=145.79.227.17:4971 , piece_id=NK55UPWQTKDWAAMPQTJPPOMZXH5VQGQ3QHJKRNMIRPFLFKQBZZIA, seg_pos={0 0}, upload=668]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo): protocol: write tcp 188.166.225.170:55424->145.79.227.17:4971: use of closed network connection; write tcp 188.166.225.170:55424->145.79.227.17:4971: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55424->145.79.227.17:4971: use of closed network connection Oct 27 15:28:49.842306875 (46m7.191614161s) [noise=true , address=145.79.227.17:4971 , piece_id=NK55UPWQTKDWAAMPQTJPPOMZXH5VQGQ3QHJKRNMIRPFLFKQBZZIA, seg_pos={0 0}, upload=668]: Long tail cancellation due to optimal threshold Oct 27 15:28:49.842335229 (46m7.191642535s) [noise=true , address=145.79.227.10:4401 , piece_id=SZNSLFKN44YVSUHRXZTF6BBVPYQODYVEBWQF3JHP6ZDA2A723WAA, seg_pos={0 0}, upload=668]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1): protocol: write tcp 188.166.225.170:50900->145.79.227.10:4401: use of closed network connection; write tcp 188.166.225.170:50900->145.79.227.10:4401: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50900->145.79.227.10:4401: use of closed network connection Oct 27 15:28:49.842355578 (46m7.191662874s) [noise=true , address=145.79.227.10:4401 , piece_id=SZNSLFKN44YVSUHRXZTF6BBVPYQODYVEBWQF3JHP6ZDA2A723WAA, seg_pos={0 0}, upload=668]: Long tail cancellation due to optimal threshold Oct 27 15:28:49.842403068 (46m7.191710354s) [noise=true , address=145.79.227.19:4079 , piece_id=QXNYHJS2SBTG2QRH7AKW7NZ6SJG7MFO5DHXCMBZHGT6PJTOZ526Q, seg_pos={0 0}, upload=668]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud): protocol: write tcp 188.166.225.170:37220->145.79.227.19:4079: use of closed network connection; write tcp 188.166.225.170:37220->145.79.227.19:4079: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37220->145.79.227.19:4079: use of closed network connection Oct 27 15:28:49.842411143 (46m7.191718419s) [noise=true , address=145.79.227.19:4079 , piece_id=QXNYHJS2SBTG2QRH7AKW7NZ6SJG7MFO5DHXCMBZHGT6PJTOZ526Q, seg_pos={0 0}, upload=668]: Long tail cancellation due to optimal threshold Oct 27 15:28:49.842428026 (46m7.191735312s) [seg_pos={0 0}, upload=668]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 27 15:28:49.842507998 (46m7.191815284s) [upload=668]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 27 15:28:49.842513478 (46m7.191820764s) [upload=668]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 27 15:28:50.170126807 (46m7.519434093s) [upload=668]: Done uploading segments... Oct 27 15:28:50.170148138 (46m7.519455424s) [upload=668]: Upload finished. err: <nil> Duration: 5.824323175s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 6.13865605s FAIL (2025-10-27 15:42:18 UTC) orbiter warning: context deadline exceeded Duration: 6.138690555s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 3/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=3) [.........X..............XX................X.................] duration: 3.733676957s FAIL (2025-10-27 15:42:07 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.733700923s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 5.694379037s FAIL (2025-10-27 15:42:03 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.694416278s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 1m0.000487629s FAIL (2025-10-27 15:41:52 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.00087183s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 6.262108331s FAIL (2025-10-27 15:41:50 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, Piece ID: 76MPB5O5XOR54TOXYCUNK2RBQMWYTWADCBAZV2KQYJ5K7XXDVFMQ) context deadline exceeded; piecestore: (Node ID: 12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, Piece ID: GR3XECVFMZ5E3HDMKK2HLKJREMIOZGWIUI7J3NSK7QGB523G7YJQ) context deadline exceeded; piecestore: (Node ID: 12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, Piece ID: FHAUWKUQAHGYR6ZJHKBCXT3ZPBG42MMJSADUGU7CIFUO3JPPNKKA) context deadline exceeded; piecestore: (Node ID: 1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV, Piece ID: 2EGSCSNCQJFXYT54PWEM33SP3RUYICBKKE7IDRQNMZ2ONPM6I5UA) context deadline exceeded ======LOGS====== ]: Done starting upload Oct 27 15:28:22.900340512 (45m40.249647808s) [upload=668]: Uploading segments... Oct 27 15:28:22.900564057 (45m40.249871343s) [upload=668]: Got next segment. Inline: false Oct 27 15:28:22.900613051 (45m40.249920337s) [upload=668]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 27 15:28:22.900620896 (45m40.249928182s) [upload=668]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 27 15:28:23.560720563 (45m40.91002784s) [seg_pos={0 0}, upload=668]: Begin upload segment... Oct 27 15:28:23.560802119 (45m40.910109405s) [seg_pos={0 0}, upload=668]: Done begin upload segment. Oct 27 15:28:23.560839660 (45m40.910146947s) [noise=true , address=145.79.227.20:4437 , piece_id=2EGSCSNCQJFXYT54PWEM33SP3RUYICBKKE7IDRQNMZ2ONPM6I5UA, seg_pos={0 0}, upload=668]: Uploading piece... Oct 27 15:28:23.560885057 (45m40.910192353s) [noise=true , address=145.79.227.19:4079 , piece_id=QXNYHJS2SBTG2QRH7AKW7NZ6SJG7MFO5DHXCMBZHGT6PJTOZ526Q, seg_pos={0 0}, upload=668]: Uploading piece... Oct 27 15:28:23.561607883 (45m40.910915169s) [noise=true , address=145.79.227.26:4653 , piece_id=MXREE4NCFOGYEHVQNFHZYD5DLD4PHBJS64JYSLTXXJZG7AC3LH6Q, seg_pos={0 0}, upload=668]: Uploading piece... Oct 27 15:28:23.561670783 (45m40.910978069s) [noise=true , address=77.90.31.10:4518 , piece_id=76MPB5O5XOR54TOXYCUNK2RBQMWYTWADCBAZV2KQYJ5K7XXDVFMQ, seg_pos={0 0}, upload=668]: Uploading piece... Oct 27 15:28:23.561966716 (45m40.911274002s) [noise=true , address=145.79.227.10:4401 , piece_id=SZNSLFKN44YVSUHRXZTF6BBVPYQODYVEBWQF3JHP6ZDA2A723WAA, seg_pos={0 0}, upload=668]: Uploading piece... Oct 27 15:28:23.562189701 (45m40.911496987s) [noise=true , address=145.79.227.4:4186 , piece_id=6APC73J3JHGVX3ZNCN3UL5DLHXRYORPQT6AYPFPX6ZQLQIRF5TNA, seg_pos={0 0}, upload=668]: Uploading piece... Oct 27 15:28:23.562277909 (45m40.911585195s) [noise=true , address=145.79.227.7:4294 , piece_id=QFLUZYLIGHO5E3ABEA5LFJPHAKDOF3J55Q4BRF5NXVKDNEHV5DBA, seg_pos={0 0}, upload=668]: Uploading piece... Oct 27 15:28:23.562572600 (45m40.911879887s) [noise=true , address=145.79.227.5:4222 , piece_id=B7W6PCNLLAPHXL7XOSJXEOVX6XZGNFDJWXR7MVYDGCONQCN55HYA, seg_pos={0 0}, upload=668]: Uploading piece... Oct 27 15:28:23.562586527 (45m40.911893813s) [noise=true , address=145.79.227.9:4366 , piece_id=5XLRE4OGUCQGZA5UXVNTNTFAMAGGQNHRR4MX23L3362K2DZLDREQ, seg_pos={0 0}, upload=668]: Uploading piece... Oct 27 15:28:23.562888732 (45m40.912196018s) [noise=true , address=145.79.227.21:4475 , piece_id=VDOFRCJKDQLXV4CBM4MA3UGK2PZMRJ3BW6H7PKJRZMZXW56Y5G4A, seg_pos={0 0}, upload=668]: Uploading piece... Oct 27 15:28:23.562953846 (45m40.912261123s) [noise=true , address=145.79.227.17:4971 , piece_id=NK55UPWQTKDWAAMPQTJPPOMZXH5VQGQ3QHJKRNMIRPFLFKQBZZIA, seg_pos={0 0}, upload=668]: Uploading piece... Oct 27 15:28:23.563182882 (45m40.912490158s) [noise=true , address=77.90.31.18:4807 , piece_id=OPDBUFUCTPZN2HH2SORHJL5L4EX2VWXATEFEIIEN5QTBQQFRGSZA, seg_pos={0 0}, upload=668]: Uploading piece... Oct 27 15:28:23.563335423 (45m40.912642709s) [noise=true , address=145.79.227.24:4582 , piece_id=K7COJQ2UG3CTGF4JVO27PTGWY4E3XGGRQWKC3LW24Y565B52YJCQ, seg_pos={0 0}, upload=668]: Uploading piece... Oct 27 15:28:23.563479036 (45m40.912786323s) [noise=true , address=77.90.31.13:4626 , piece_id=GR3XECVFMZ5E3HDMKK2HLKJREMIOZGWIUI7J3NSK7QGB523G7YJQ, seg_pos={0 0}, upload=668]: Uploading piece... Oct 27 15:28:23.563711468 (45m40.913018754s) [noise=true , address=45.91.116.5:4298 , piece_id=ATXRIPPIX6BAHUXCENJVDJ25K2Z6UH3QYL4MTBWPZKP7LFGHBX6A, seg_pos={0 0}, upload=668]: Uploading piece... Oct 27 15:28:23.564137260 (45m40.913444546s) [noise=true , address=145.79.227.5:4223 , piece_id=FHAUWKUQAHGYR6ZJHKBCXT3ZPBG42MMJSADUGU7CIFUO3JPPNKKA, seg_pos={0 0}, upload=668]: Uploading piece... Oct 27 15:28:27.241369967 (45m44.590677243s) [upload=668]: Next returned nil segment Oct 27 15:28:27.241385446 (45m44.590692732s) [upload=668]: Waiting for error group managing segments... Oct 27 15:28:29.744963349 (45m47.094270636s) [noise=true , address=145.79.227.5:4222 , piece_id=B7W6PCNLLAPHXL7XOSJXEOVX6XZGNFDJWXR7MVYDGCONQCN55HYA, seg_pos={0 0}, upload=668]: Done uploading piece. err: <nil> Oct 27 15:28:29.821202504 (45m47.17050978s) [noise=true , address=145.79.227.21:4475 , piece_id=VDOFRCJKDQLXV4CBM4MA3UGK2PZMRJ3BW6H7PKJRZMZXW56Y5G4A, seg_pos={0 0}, upload=668]: Done uploading piece. err: <nil> Oct 27 15:28:32.126653324 (45m49.47596061s) [noise=true , address=77.90.31.18:4807 , piece_id=OPDBUFUCTPZN2HH2SORHJL5L4EX2VWXATEFEIIEN5QTBQQFRGSZA, seg_pos={0 0}, upload=668]: Done uploading piece. err: <nil> Oct 27 15:28:33.087272849 (45m50.436580125s) [noise=true , address=45.91.116.5:4298 , piece_id=ATXRIPPIX6BAHUXCENJVDJ25K2Z6UH3QYL4MTBWPZKP7LFGHBX6A, seg_pos={0 0}, upload=668]: Done uploading piece. err: <nil> Oct 27 15:28:38.581899084 (45m55.93120637s) [noise=true , address=145.79.227.26:4653 , piece_id=MXREE4NCFOGYEHVQNFHZYD5DLD4PHBJS64JYSLTXXJZG7AC3LH6Q, seg_pos={0 0}, upload=668]: Done uploading piece. err: <nil> Oct 27 15:28:39.053322174 (45m56.40262946s) [noise=true , address=145.79.227.24:4582 , piece_id=K7COJQ2UG3CTGF4JVO27PTGWY4E3XGGRQWKC3LW24Y565B52YJCQ, seg_pos={0 0}, upload=668]: Done uploading piece. err: <nil> Oct 27 15:28:39.717124420 (45m57.066431696s) [noise=true , address=145.79.227.20:4437 , piece_id=2EGSCSNCQJFXYT54PWEM33SP3RUYICBKKE7IDRQNMZ2ONPM6I5UA, seg_pos={0 0}, upload=668]: Done uploading piece. err: <nil> Oct 27 15:28:41.618415745 (45m58.967723031s) [noise=true , address=77.90.31.13:4626 , piece_id=GR3XECVFMZ5E3HDMKK2HLKJREMIOZGWIUI7J3NSK7QGB523G7YJQ, seg_pos={0 0}, upload=668]: Done uploading piece. err: <nil> Oct 27 15:28:42.068020890 (45m59.417328166s) [noise=true , address=145.79.227.5:4223 , piece_id=FHAUWKUQAHGYR6ZJHKBCXT3ZPBG42MMJSADUGU7CIFUO3JPPNKKA, seg_pos={0 0}, upload=668]: Done uploading piece. err: <nil> Oct 27 15:28:42.150893235 (45m59.500200521s) [noise=true , address=77.90.31.10:4518 , piece_id=76MPB5O5XOR54TOXYCUNK2RBQMWYTWADCBAZV2KQYJ5K7XXDVFMQ, seg_pos={0 0}, upload=668]: Done uploading piece. err: <nil> Oct 27 15:28:43.703285619 (46m1.052592905s) [noise=true , address=145.79.227.9:4366 , piece_id=5XLRE4OGUCQGZA5UXVNTNTFAMAGGQNHRR4MX23L3362K2DZLDREQ, seg_pos={0 0}, upload=668]: Done uploading piece. err: <nil> Oct 27 15:28:49.699904003 (46m7.049211289s) [noise=true , address=145.79.227.4:4186 , piece_id=6APC73J3JHGVX3ZNCN3UL5DLHXRYORPQT6AYPFPX6ZQLQIRF5TNA, seg_pos={0 0}, upload=668]: Done uploading piece. err: <nil> Oct 27 15:28:49.842045738 (46m7.191353014s) [noise=true , address=145.79.227.7:4294 , piece_id=QFLUZYLIGHO5E3ABEA5LFJPHAKDOF3J55Q4BRF5NXVKDNEHV5DBA, seg_pos={0 0}, upload=668]: Done uploading piece. err: <nil> Oct 27 15:28:49.842081656 (46m7.191388932s) [seg_pos={0 0}, upload=668]: Segment reached optimal threshold of 13 pieces. Oct 27 15:28:49.842296495 (46m7.191603781s) [noise=true , address=145.79.227.17:4971 , piece_id=NK55UPWQTKDWAAMPQTJPPOMZXH5VQGQ3QHJKRNMIRPFLFKQBZZIA, seg_pos={0 0}, upload=668]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo): protocol: write tcp 188.166.225.170:55424->145.79.227.17:4971: use of closed network connection; write tcp 188.166.225.170:55424->145.79.227.17:4971: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55424->145.79.227.17:4971: use of closed network connection Oct 27 15:28:49.842306875 (46m7.191614161s) [noise=true , address=145.79.227.17:4971 , piece_id=NK55UPWQTKDWAAMPQTJPPOMZXH5VQGQ3QHJKRNMIRPFLFKQBZZIA, seg_pos={0 0}, upload=668]: Long tail cancellation due to optimal threshold Oct 27 15:28:49.842335229 (46m7.191642535s) [noise=true , address=145.79.227.10:4401 , piece_id=SZNSLFKN44YVSUHRXZTF6BBVPYQODYVEBWQF3JHP6ZDA2A723WAA, seg_pos={0 0}, upload=668]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1): protocol: write tcp 188.166.225.170:50900->145.79.227.10:4401: use of closed network connection; write tcp 188.166.225.170:50900->145.79.227.10:4401: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50900->145.79.227.10:4401: use of closed network connection Oct 27 15:28:49.842355578 (46m7.191662874s) [noise=true , address=145.79.227.10:4401 , piece_id=SZNSLFKN44YVSUHRXZTF6BBVPYQODYVEBWQF3JHP6ZDA2A723WAA, seg_pos={0 0}, upload=668]: Long tail cancellation due to optimal threshold Oct 27 15:28:49.842403068 (46m7.191710354s) [noise=true , address=145.79.227.19:4079 , piece_id=QXNYHJS2SBTG2QRH7AKW7NZ6SJG7MFO5DHXCMBZHGT6PJTOZ526Q, seg_pos={0 0}, upload=668]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud): protocol: write tcp 188.166.225.170:37220->145.79.227.19:4079: use of closed network connection; write tcp 188.166.225.170:37220->145.79.227.19:4079: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37220->145.79.227.19:4079: use of closed network connection Oct 27 15:28:49.842411143 (46m7.191718419s) [noise=true , address=145.79.227.19:4079 , piece_id=QXNYHJS2SBTG2QRH7AKW7NZ6SJG7MFO5DHXCMBZHGT6PJTOZ526Q, seg_pos={0 0}, upload=668]: Long tail cancellation due to optimal threshold Oct 27 15:28:49.842428026 (46m7.191735312s) [seg_pos={0 0}, upload=668]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 27 15:28:49.842507998 (46m7.191815284s) [upload=668]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 27 15:28:49.842513478 (46m7.191820764s) [upload=668]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 27 15:28:50.170126807 (46m7.519434093s) [upload=668]: Done uploading segments... Oct 27 15:28:50.170148138 (46m7.519455424s) [upload=668]: Upload finished. err: <nil> Duration: 6.26214481s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 6.076075684s FAIL (2025-10-27 15:41:16 UTC) orbiter warning: context deadline exceeded Duration: 6.076097865s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 3/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=3) [..........X..............XX................X................] duration: 3.415307138s FAIL (2025-10-27 15:41:14 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.415400625s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 6.119027811s FAIL (2025-10-27 15:40:39 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, Piece ID: 76MPB5O5XOR54TOXYCUNK2RBQMWYTWADCBAZV2KQYJ5K7XXDVFMQ) context deadline exceeded; piecestore: (Node ID: 12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, Piece ID: GR3XECVFMZ5E3HDMKK2HLKJREMIOZGWIUI7J3NSK7QGB523G7YJQ) context deadline exceeded; piecestore: (Node ID: 12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, Piece ID: FHAUWKUQAHGYR6ZJHKBCXT3ZPBG42MMJSADUGU7CIFUO3JPPNKKA) context deadline exceeded; piecestore: (Node ID: 1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV, Piece ID: 2EGSCSNCQJFXYT54PWEM33SP3RUYICBKKE7IDRQNMZ2ONPM6I5UA) context deadline exceeded ======LOGS====== ]: Done starting upload Oct 27 15:28:22.900340512 (45m40.249647808s) [upload=668]: Uploading segments... Oct 27 15:28:22.900564057 (45m40.249871343s) [upload=668]: Got next segment. Inline: false Oct 27 15:28:22.900613051 (45m40.249920337s) [upload=668]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 27 15:28:22.900620896 (45m40.249928182s) [upload=668]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 27 15:28:23.560720563 (45m40.91002784s) [seg_pos={0 0}, upload=668]: Begin upload segment... Oct 27 15:28:23.560802119 (45m40.910109405s) [seg_pos={0 0}, upload=668]: Done begin upload segment. Oct 27 15:28:23.560839660 (45m40.910146947s) [noise=true , address=145.79.227.20:4437 , piece_id=2EGSCSNCQJFXYT54PWEM33SP3RUYICBKKE7IDRQNMZ2ONPM6I5UA, seg_pos={0 0}, upload=668]: Uploading piece... Oct 27 15:28:23.560885057 (45m40.910192353s) [noise=true , address=145.79.227.19:4079 , piece_id=QXNYHJS2SBTG2QRH7AKW7NZ6SJG7MFO5DHXCMBZHGT6PJTOZ526Q, seg_pos={0 0}, upload=668]: Uploading piece... Oct 27 15:28:23.561607883 (45m40.910915169s) [noise=true , address=145.79.227.26:4653 , piece_id=MXREE4NCFOGYEHVQNFHZYD5DLD4PHBJS64JYSLTXXJZG7AC3LH6Q, seg_pos={0 0}, upload=668]: Uploading piece... Oct 27 15:28:23.561670783 (45m40.910978069s) [noise=true , address=77.90.31.10:4518 , piece_id=76MPB5O5XOR54TOXYCUNK2RBQMWYTWADCBAZV2KQYJ5K7XXDVFMQ, seg_pos={0 0}, upload=668]: Uploading piece... Oct 27 15:28:23.561966716 (45m40.911274002s) [noise=true , address=145.79.227.10:4401 , piece_id=SZNSLFKN44YVSUHRXZTF6BBVPYQODYVEBWQF3JHP6ZDA2A723WAA, seg_pos={0 0}, upload=668]: Uploading piece... Oct 27 15:28:23.562189701 (45m40.911496987s) [noise=true , address=145.79.227.4:4186 , piece_id=6APC73J3JHGVX3ZNCN3UL5DLHXRYORPQT6AYPFPX6ZQLQIRF5TNA, seg_pos={0 0}, upload=668]: Uploading piece... Oct 27 15:28:23.562277909 (45m40.911585195s) [noise=true , address=145.79.227.7:4294 , piece_id=QFLUZYLIGHO5E3ABEA5LFJPHAKDOF3J55Q4BRF5NXVKDNEHV5DBA, seg_pos={0 0}, upload=668]: Uploading piece... Oct 27 15:28:23.562572600 (45m40.911879887s) [noise=true , address=145.79.227.5:4222 , piece_id=B7W6PCNLLAPHXL7XOSJXEOVX6XZGNFDJWXR7MVYDGCONQCN55HYA, seg_pos={0 0}, upload=668]: Uploading piece... Oct 27 15:28:23.562586527 (45m40.911893813s) [noise=true , address=145.79.227.9:4366 , piece_id=5XLRE4OGUCQGZA5UXVNTNTFAMAGGQNHRR4MX23L3362K2DZLDREQ, seg_pos={0 0}, upload=668]: Uploading piece... Oct 27 15:28:23.562888732 (45m40.912196018s) [noise=true , address=145.79.227.21:4475 , piece_id=VDOFRCJKDQLXV4CBM4MA3UGK2PZMRJ3BW6H7PKJRZMZXW56Y5G4A, seg_pos={0 0}, upload=668]: Uploading piece... Oct 27 15:28:23.562953846 (45m40.912261123s) [noise=true , address=145.79.227.17:4971 , piece_id=NK55UPWQTKDWAAMPQTJPPOMZXH5VQGQ3QHJKRNMIRPFLFKQBZZIA, seg_pos={0 0}, upload=668]: Uploading piece... Oct 27 15:28:23.563182882 (45m40.912490158s) [noise=true , address=77.90.31.18:4807 , piece_id=OPDBUFUCTPZN2HH2SORHJL5L4EX2VWXATEFEIIEN5QTBQQFRGSZA, seg_pos={0 0}, upload=668]: Uploading piece... Oct 27 15:28:23.563335423 (45m40.912642709s) [noise=true , address=145.79.227.24:4582 , piece_id=K7COJQ2UG3CTGF4JVO27PTGWY4E3XGGRQWKC3LW24Y565B52YJCQ, seg_pos={0 0}, upload=668]: Uploading piece... Oct 27 15:28:23.563479036 (45m40.912786323s) [noise=true , address=77.90.31.13:4626 , piece_id=GR3XECVFMZ5E3HDMKK2HLKJREMIOZGWIUI7J3NSK7QGB523G7YJQ, seg_pos={0 0}, upload=668]: Uploading piece... Oct 27 15:28:23.563711468 (45m40.913018754s) [noise=true , address=45.91.116.5:4298 , piece_id=ATXRIPPIX6BAHUXCENJVDJ25K2Z6UH3QYL4MTBWPZKP7LFGHBX6A, seg_pos={0 0}, upload=668]: Uploading piece... Oct 27 15:28:23.564137260 (45m40.913444546s) [noise=true , address=145.79.227.5:4223 , piece_id=FHAUWKUQAHGYR6ZJHKBCXT3ZPBG42MMJSADUGU7CIFUO3JPPNKKA, seg_pos={0 0}, upload=668]: Uploading piece... Oct 27 15:28:27.241369967 (45m44.590677243s) [upload=668]: Next returned nil segment Oct 27 15:28:27.241385446 (45m44.590692732s) [upload=668]: Waiting for error group managing segments... Oct 27 15:28:29.744963349 (45m47.094270636s) [noise=true , address=145.79.227.5:4222 , piece_id=B7W6PCNLLAPHXL7XOSJXEOVX6XZGNFDJWXR7MVYDGCONQCN55HYA, seg_pos={0 0}, upload=668]: Done uploading piece. err: <nil> Oct 27 15:28:29.821202504 (45m47.17050978s) [noise=true , address=145.79.227.21:4475 , piece_id=VDOFRCJKDQLXV4CBM4MA3UGK2PZMRJ3BW6H7PKJRZMZXW56Y5G4A, seg_pos={0 0}, upload=668]: Done uploading piece. err: <nil> Oct 27 15:28:32.126653324 (45m49.47596061s) [noise=true , address=77.90.31.18:4807 , piece_id=OPDBUFUCTPZN2HH2SORHJL5L4EX2VWXATEFEIIEN5QTBQQFRGSZA, seg_pos={0 0}, upload=668]: Done uploading piece. err: <nil> Oct 27 15:28:33.087272849 (45m50.436580125s) [noise=true , address=45.91.116.5:4298 , piece_id=ATXRIPPIX6BAHUXCENJVDJ25K2Z6UH3QYL4MTBWPZKP7LFGHBX6A, seg_pos={0 0}, upload=668]: Done uploading piece. err: <nil> Oct 27 15:28:38.581899084 (45m55.93120637s) [noise=true , address=145.79.227.26:4653 , piece_id=MXREE4NCFOGYEHVQNFHZYD5DLD4PHBJS64JYSLTXXJZG7AC3LH6Q, seg_pos={0 0}, upload=668]: Done uploading piece. err: <nil> Oct 27 15:28:39.053322174 (45m56.40262946s) [noise=true , address=145.79.227.24:4582 , piece_id=K7COJQ2UG3CTGF4JVO27PTGWY4E3XGGRQWKC3LW24Y565B52YJCQ, seg_pos={0 0}, upload=668]: Done uploading piece. err: <nil> Oct 27 15:28:39.717124420 (45m57.066431696s) [noise=true , address=145.79.227.20:4437 , piece_id=2EGSCSNCQJFXYT54PWEM33SP3RUYICBKKE7IDRQNMZ2ONPM6I5UA, seg_pos={0 0}, upload=668]: Done uploading piece. err: <nil> Oct 27 15:28:41.618415745 (45m58.967723031s) [noise=true , address=77.90.31.13:4626 , piece_id=GR3XECVFMZ5E3HDMKK2HLKJREMIOZGWIUI7J3NSK7QGB523G7YJQ, seg_pos={0 0}, upload=668]: Done uploading piece. err: <nil> Oct 27 15:28:42.068020890 (45m59.417328166s) [noise=true , address=145.79.227.5:4223 , piece_id=FHAUWKUQAHGYR6ZJHKBCXT3ZPBG42MMJSADUGU7CIFUO3JPPNKKA, seg_pos={0 0}, upload=668]: Done uploading piece. err: <nil> Oct 27 15:28:42.150893235 (45m59.500200521s) [noise=true , address=77.90.31.10:4518 , piece_id=76MPB5O5XOR54TOXYCUNK2RBQMWYTWADCBAZV2KQYJ5K7XXDVFMQ, seg_pos={0 0}, upload=668]: Done uploading piece. err: <nil> Oct 27 15:28:43.703285619 (46m1.052592905s) [noise=true , address=145.79.227.9:4366 , piece_id=5XLRE4OGUCQGZA5UXVNTNTFAMAGGQNHRR4MX23L3362K2DZLDREQ, seg_pos={0 0}, upload=668]: Done uploading piece. err: <nil> Oct 27 15:28:49.699904003 (46m7.049211289s) [noise=true , address=145.79.227.4:4186 , piece_id=6APC73J3JHGVX3ZNCN3UL5DLHXRYORPQT6AYPFPX6ZQLQIRF5TNA, seg_pos={0 0}, upload=668]: Done uploading piece. err: <nil> Oct 27 15:28:49.842045738 (46m7.191353014s) [noise=true , address=145.79.227.7:4294 , piece_id=QFLUZYLIGHO5E3ABEA5LFJPHAKDOF3J55Q4BRF5NXVKDNEHV5DBA, seg_pos={0 0}, upload=668]: Done uploading piece. err: <nil> Oct 27 15:28:49.842081656 (46m7.191388932s) [seg_pos={0 0}, upload=668]: Segment reached optimal threshold of 13 pieces. Oct 27 15:28:49.842296495 (46m7.191603781s) [noise=true , address=145.79.227.17:4971 , piece_id=NK55UPWQTKDWAAMPQTJPPOMZXH5VQGQ3QHJKRNMIRPFLFKQBZZIA, seg_pos={0 0}, upload=668]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo): protocol: write tcp 188.166.225.170:55424->145.79.227.17:4971: use of closed network connection; write tcp 188.166.225.170:55424->145.79.227.17:4971: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55424->145.79.227.17:4971: use of closed network connection Oct 27 15:28:49.842306875 (46m7.191614161s) [noise=true , address=145.79.227.17:4971 , piece_id=NK55UPWQTKDWAAMPQTJPPOMZXH5VQGQ3QHJKRNMIRPFLFKQBZZIA, seg_pos={0 0}, upload=668]: Long tail cancellation due to optimal threshold Oct 27 15:28:49.842335229 (46m7.191642535s) [noise=true , address=145.79.227.10:4401 , piece_id=SZNSLFKN44YVSUHRXZTF6BBVPYQODYVEBWQF3JHP6ZDA2A723WAA, seg_pos={0 0}, upload=668]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1): protocol: write tcp 188.166.225.170:50900->145.79.227.10:4401: use of closed network connection; write tcp 188.166.225.170:50900->145.79.227.10:4401: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50900->145.79.227.10:4401: use of closed network connection Oct 27 15:28:49.842355578 (46m7.191662874s) [noise=true , address=145.79.227.10:4401 , piece_id=SZNSLFKN44YVSUHRXZTF6BBVPYQODYVEBWQF3JHP6ZDA2A723WAA, seg_pos={0 0}, upload=668]: Long tail cancellation due to optimal threshold Oct 27 15:28:49.842403068 (46m7.191710354s) [noise=true , address=145.79.227.19:4079 , piece_id=QXNYHJS2SBTG2QRH7AKW7NZ6SJG7MFO5DHXCMBZHGT6PJTOZ526Q, seg_pos={0 0}, upload=668]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud): protocol: write tcp 188.166.225.170:37220->145.79.227.19:4079: use of closed network connection; write tcp 188.166.225.170:37220->145.79.227.19:4079: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37220->145.79.227.19:4079: use of closed network connection Oct 27 15:28:49.842411143 (46m7.191718419s) [noise=true , address=145.79.227.19:4079 , piece_id=QXNYHJS2SBTG2QRH7AKW7NZ6SJG7MFO5DHXCMBZHGT6PJTOZ526Q, seg_pos={0 0}, upload=668]: Long tail cancellation due to optimal threshold Oct 27 15:28:49.842428026 (46m7.191735312s) [seg_pos={0 0}, upload=668]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 27 15:28:49.842507998 (46m7.191815284s) [upload=668]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 27 15:28:49.842513478 (46m7.191820764s) [upload=668]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 27 15:28:50.170126807 (46m7.519434093s) [upload=668]: Done uploading segments... Oct 27 15:28:50.170148138 (46m7.519455424s) [upload=668]: Upload finished. err: <nil> Duration: 6.119061345s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 6.342317797s FAIL (2025-10-27 15:40:33 UTC) orbiter warning: context deadline exceeded Duration: 6.342343535s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 3/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=3) [...........X..............XX................X...............] duration: 3.437861839s FAIL (2025-10-27 15:40:13 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.437888439s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 3/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=3) [............X..............XX................X..............] duration: 3.37713641s FAIL (2025-10-27 15:39:51 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.377172948s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 6.115746921s FAIL (2025-10-27 15:39:33 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, Piece ID: 76MPB5O5XOR54TOXYCUNK2RBQMWYTWADCBAZV2KQYJ5K7XXDVFMQ) context deadline exceeded; piecestore: (Node ID: 12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, Piece ID: GR3XECVFMZ5E3HDMKK2HLKJREMIOZGWIUI7J3NSK7QGB523G7YJQ) context deadline exceeded; piecestore: (Node ID: 12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, Piece ID: FHAUWKUQAHGYR6ZJHKBCXT3ZPBG42MMJSADUGU7CIFUO3JPPNKKA) context deadline exceeded; piecestore: (Node ID: 1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV, Piece ID: 2EGSCSNCQJFXYT54PWEM33SP3RUYICBKKE7IDRQNMZ2ONPM6I5UA) context deadline exceeded ======LOGS====== ]: Done starting upload Oct 27 15:28:22.900340512 (45m40.249647808s) [upload=668]: Uploading segments... Oct 27 15:28:22.900564057 (45m40.249871343s) [upload=668]: Got next segment. Inline: false Oct 27 15:28:22.900613051 (45m40.249920337s) [upload=668]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 27 15:28:22.900620896 (45m40.249928182s) [upload=668]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 27 15:28:23.560720563 (45m40.91002784s) [seg_pos={0 0}, upload=668]: Begin upload segment... Oct 27 15:28:23.560802119 (45m40.910109405s) [seg_pos={0 0}, upload=668]: Done begin upload segment. Oct 27 15:28:23.560839660 (45m40.910146947s) [noise=true , address=145.79.227.20:4437 , piece_id=2EGSCSNCQJFXYT54PWEM33SP3RUYICBKKE7IDRQNMZ2ONPM6I5UA, seg_pos={0 0}, upload=668]: Uploading piece... Oct 27 15:28:23.560885057 (45m40.910192353s) [noise=true , address=145.79.227.19:4079 , piece_id=QXNYHJS2SBTG2QRH7AKW7NZ6SJG7MFO5DHXCMBZHGT6PJTOZ526Q, seg_pos={0 0}, upload=668]: Uploading piece... Oct 27 15:28:23.561607883 (45m40.910915169s) [noise=true , address=145.79.227.26:4653 , piece_id=MXREE4NCFOGYEHVQNFHZYD5DLD4PHBJS64JYSLTXXJZG7AC3LH6Q, seg_pos={0 0}, upload=668]: Uploading piece... Oct 27 15:28:23.561670783 (45m40.910978069s) [noise=true , address=77.90.31.10:4518 , piece_id=76MPB5O5XOR54TOXYCUNK2RBQMWYTWADCBAZV2KQYJ5K7XXDVFMQ, seg_pos={0 0}, upload=668]: Uploading piece... Oct 27 15:28:23.561966716 (45m40.911274002s) [noise=true , address=145.79.227.10:4401 , piece_id=SZNSLFKN44YVSUHRXZTF6BBVPYQODYVEBWQF3JHP6ZDA2A723WAA, seg_pos={0 0}, upload=668]: Uploading piece... Oct 27 15:28:23.562189701 (45m40.911496987s) [noise=true , address=145.79.227.4:4186 , piece_id=6APC73J3JHGVX3ZNCN3UL5DLHXRYORPQT6AYPFPX6ZQLQIRF5TNA, seg_pos={0 0}, upload=668]: Uploading piece... Oct 27 15:28:23.562277909 (45m40.911585195s) [noise=true , address=145.79.227.7:4294 , piece_id=QFLUZYLIGHO5E3ABEA5LFJPHAKDOF3J55Q4BRF5NXVKDNEHV5DBA, seg_pos={0 0}, upload=668]: Uploading piece... Oct 27 15:28:23.562572600 (45m40.911879887s) [noise=true , address=145.79.227.5:4222 , piece_id=B7W6PCNLLAPHXL7XOSJXEOVX6XZGNFDJWXR7MVYDGCONQCN55HYA, seg_pos={0 0}, upload=668]: Uploading piece... Oct 27 15:28:23.562586527 (45m40.911893813s) [noise=true , address=145.79.227.9:4366 , piece_id=5XLRE4OGUCQGZA5UXVNTNTFAMAGGQNHRR4MX23L3362K2DZLDREQ, seg_pos={0 0}, upload=668]: Uploading piece... Oct 27 15:28:23.562888732 (45m40.912196018s) [noise=true , address=145.79.227.21:4475 , piece_id=VDOFRCJKDQLXV4CBM4MA3UGK2PZMRJ3BW6H7PKJRZMZXW56Y5G4A, seg_pos={0 0}, upload=668]: Uploading piece... Oct 27 15:28:23.562953846 (45m40.912261123s) [noise=true , address=145.79.227.17:4971 , piece_id=NK55UPWQTKDWAAMPQTJPPOMZXH5VQGQ3QHJKRNMIRPFLFKQBZZIA, seg_pos={0 0}, upload=668]: Uploading piece... Oct 27 15:28:23.563182882 (45m40.912490158s) [noise=true , address=77.90.31.18:4807 , piece_id=OPDBUFUCTPZN2HH2SORHJL5L4EX2VWXATEFEIIEN5QTBQQFRGSZA, seg_pos={0 0}, upload=668]: Uploading piece... Oct 27 15:28:23.563335423 (45m40.912642709s) [noise=true , address=145.79.227.24:4582 , piece_id=K7COJQ2UG3CTGF4JVO27PTGWY4E3XGGRQWKC3LW24Y565B52YJCQ, seg_pos={0 0}, upload=668]: Uploading piece... Oct 27 15:28:23.563479036 (45m40.912786323s) [noise=true , address=77.90.31.13:4626 , piece_id=GR3XECVFMZ5E3HDMKK2HLKJREMIOZGWIUI7J3NSK7QGB523G7YJQ, seg_pos={0 0}, upload=668]: Uploading piece... Oct 27 15:28:23.563711468 (45m40.913018754s) [noise=true , address=45.91.116.5:4298 , piece_id=ATXRIPPIX6BAHUXCENJVDJ25K2Z6UH3QYL4MTBWPZKP7LFGHBX6A, seg_pos={0 0}, upload=668]: Uploading piece... Oct 27 15:28:23.564137260 (45m40.913444546s) [noise=true , address=145.79.227.5:4223 , piece_id=FHAUWKUQAHGYR6ZJHKBCXT3ZPBG42MMJSADUGU7CIFUO3JPPNKKA, seg_pos={0 0}, upload=668]: Uploading piece... Oct 27 15:28:27.241369967 (45m44.590677243s) [upload=668]: Next returned nil segment Oct 27 15:28:27.241385446 (45m44.590692732s) [upload=668]: Waiting for error group managing segments... Oct 27 15:28:29.744963349 (45m47.094270636s) [noise=true , address=145.79.227.5:4222 , piece_id=B7W6PCNLLAPHXL7XOSJXEOVX6XZGNFDJWXR7MVYDGCONQCN55HYA, seg_pos={0 0}, upload=668]: Done uploading piece. err: <nil> Oct 27 15:28:29.821202504 (45m47.17050978s) [noise=true , address=145.79.227.21:4475 , piece_id=VDOFRCJKDQLXV4CBM4MA3UGK2PZMRJ3BW6H7PKJRZMZXW56Y5G4A, seg_pos={0 0}, upload=668]: Done uploading piece. err: <nil> Oct 27 15:28:32.126653324 (45m49.47596061s) [noise=true , address=77.90.31.18:4807 , piece_id=OPDBUFUCTPZN2HH2SORHJL5L4EX2VWXATEFEIIEN5QTBQQFRGSZA, seg_pos={0 0}, upload=668]: Done uploading piece. err: <nil> Oct 27 15:28:33.087272849 (45m50.436580125s) [noise=true , address=45.91.116.5:4298 , piece_id=ATXRIPPIX6BAHUXCENJVDJ25K2Z6UH3QYL4MTBWPZKP7LFGHBX6A, seg_pos={0 0}, upload=668]: Done uploading piece. err: <nil> Oct 27 15:28:38.581899084 (45m55.93120637s) [noise=true , address=145.79.227.26:4653 , piece_id=MXREE4NCFOGYEHVQNFHZYD5DLD4PHBJS64JYSLTXXJZG7AC3LH6Q, seg_pos={0 0}, upload=668]: Done uploading piece. err: <nil> Oct 27 15:28:39.053322174 (45m56.40262946s) [noise=true , address=145.79.227.24:4582 , piece_id=K7COJQ2UG3CTGF4JVO27PTGWY4E3XGGRQWKC3LW24Y565B52YJCQ, seg_pos={0 0}, upload=668]: Done uploading piece. err: <nil> Oct 27 15:28:39.717124420 (45m57.066431696s) [noise=true , address=145.79.227.20:4437 , piece_id=2EGSCSNCQJFXYT54PWEM33SP3RUYICBKKE7IDRQNMZ2ONPM6I5UA, seg_pos={0 0}, upload=668]: Done uploading piece. err: <nil> Oct 27 15:28:41.618415745 (45m58.967723031s) [noise=true , address=77.90.31.13:4626 , piece_id=GR3XECVFMZ5E3HDMKK2HLKJREMIOZGWIUI7J3NSK7QGB523G7YJQ, seg_pos={0 0}, upload=668]: Done uploading piece. err: <nil> Oct 27 15:28:42.068020890 (45m59.417328166s) [noise=true , address=145.79.227.5:4223 , piece_id=FHAUWKUQAHGYR6ZJHKBCXT3ZPBG42MMJSADUGU7CIFUO3JPPNKKA, seg_pos={0 0}, upload=668]: Done uploading piece. err: <nil> Oct 27 15:28:42.150893235 (45m59.500200521s) [noise=true , address=77.90.31.10:4518 , piece_id=76MPB5O5XOR54TOXYCUNK2RBQMWYTWADCBAZV2KQYJ5K7XXDVFMQ, seg_pos={0 0}, upload=668]: Done uploading piece. err: <nil> Oct 27 15:28:43.703285619 (46m1.052592905s) [noise=true , address=145.79.227.9:4366 , piece_id=5XLRE4OGUCQGZA5UXVNTNTFAMAGGQNHRR4MX23L3362K2DZLDREQ, seg_pos={0 0}, upload=668]: Done uploading piece. err: <nil> Oct 27 15:28:49.699904003 (46m7.049211289s) [noise=true , address=145.79.227.4:4186 , piece_id=6APC73J3JHGVX3ZNCN3UL5DLHXRYORPQT6AYPFPX6ZQLQIRF5TNA, seg_pos={0 0}, upload=668]: Done uploading piece. err: <nil> Oct 27 15:28:49.842045738 (46m7.191353014s) [noise=true , address=145.79.227.7:4294 , piece_id=QFLUZYLIGHO5E3ABEA5LFJPHAKDOF3J55Q4BRF5NXVKDNEHV5DBA, seg_pos={0 0}, upload=668]: Done uploading piece. err: <nil> Oct 27 15:28:49.842081656 (46m7.191388932s) [seg_pos={0 0}, upload=668]: Segment reached optimal threshold of 13 pieces. Oct 27 15:28:49.842296495 (46m7.191603781s) [noise=true , address=145.79.227.17:4971 , piece_id=NK55UPWQTKDWAAMPQTJPPOMZXH5VQGQ3QHJKRNMIRPFLFKQBZZIA, seg_pos={0 0}, upload=668]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo): protocol: write tcp 188.166.225.170:55424->145.79.227.17:4971: use of closed network connection; write tcp 188.166.225.170:55424->145.79.227.17:4971: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55424->145.79.227.17:4971: use of closed network connection Oct 27 15:28:49.842306875 (46m7.191614161s) [noise=true , address=145.79.227.17:4971 , piece_id=NK55UPWQTKDWAAMPQTJPPOMZXH5VQGQ3QHJKRNMIRPFLFKQBZZIA, seg_pos={0 0}, upload=668]: Long tail cancellation due to optimal threshold Oct 27 15:28:49.842335229 (46m7.191642535s) [noise=true , address=145.79.227.10:4401 , piece_id=SZNSLFKN44YVSUHRXZTF6BBVPYQODYVEBWQF3JHP6ZDA2A723WAA, seg_pos={0 0}, upload=668]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1): protocol: write tcp 188.166.225.170:50900->145.79.227.10:4401: use of closed network connection; write tcp 188.166.225.170:50900->145.79.227.10:4401: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50900->145.79.227.10:4401: use of closed network connection Oct 27 15:28:49.842355578 (46m7.191662874s) [noise=true , address=145.79.227.10:4401 , piece_id=SZNSLFKN44YVSUHRXZTF6BBVPYQODYVEBWQF3JHP6ZDA2A723WAA, seg_pos={0 0}, upload=668]: Long tail cancellation due to optimal threshold Oct 27 15:28:49.842403068 (46m7.191710354s) [noise=true , address=145.79.227.19:4079 , piece_id=QXNYHJS2SBTG2QRH7AKW7NZ6SJG7MFO5DHXCMBZHGT6PJTOZ526Q, seg_pos={0 0}, upload=668]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud): protocol: write tcp 188.166.225.170:37220->145.79.227.19:4079: use of closed network connection; write tcp 188.166.225.170:37220->145.79.227.19:4079: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37220->145.79.227.19:4079: use of closed network connection Oct 27 15:28:49.842411143 (46m7.191718419s) [noise=true , address=145.79.227.19:4079 , piece_id=QXNYHJS2SBTG2QRH7AKW7NZ6SJG7MFO5DHXCMBZHGT6PJTOZ526Q, seg_pos={0 0}, upload=668]: Long tail cancellation due to optimal threshold Oct 27 15:28:49.842428026 (46m7.191735312s) [seg_pos={0 0}, upload=668]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 27 15:28:49.842507998 (46m7.191815284s) [upload=668]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 27 15:28:49.842513478 (46m7.191820764s) [upload=668]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 27 15:28:50.170126807 (46m7.519434093s) [upload=668]: Done uploading segments... Oct 27 15:28:50.170148138 (46m7.519455424s) [upload=668]: Upload finished. err: <nil> Duration: 6.115775946s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 3/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=3) [.............X..............XX................X.............] duration: 3.350533915s FAIL (2025-10-27 15:39:33 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.351311887s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 6.386489025s FAIL (2025-10-27 15:39:19 UTC) orbiter warning: context deadline exceeded Duration: 6.386532948s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 6.447302207s FAIL (2025-10-27 15:39:13 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, Piece ID: 76MPB5O5XOR54TOXYCUNK2RBQMWYTWADCBAZV2KQYJ5K7XXDVFMQ) context deadline exceeded; piecestore: (Node ID: 12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, Piece ID: GR3XECVFMZ5E3HDMKK2HLKJREMIOZGWIUI7J3NSK7QGB523G7YJQ) context deadline exceeded; piecestore: (Node ID: 12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, Piece ID: FHAUWKUQAHGYR6ZJHKBCXT3ZPBG42MMJSADUGU7CIFUO3JPPNKKA) context deadline exceeded; piecestore: (Node ID: 1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV, Piece ID: 2EGSCSNCQJFXYT54PWEM33SP3RUYICBKKE7IDRQNMZ2ONPM6I5UA) context deadline exceeded ======LOGS====== ]: Done starting upload Oct 27 15:28:22.900340512 (45m40.249647808s) [upload=668]: Uploading segments... Oct 27 15:28:22.900564057 (45m40.249871343s) [upload=668]: Got next segment. Inline: false Oct 27 15:28:22.900613051 (45m40.249920337s) [upload=668]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 27 15:28:22.900620896 (45m40.249928182s) [upload=668]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 27 15:28:23.560720563 (45m40.91002784s) [seg_pos={0 0}, upload=668]: Begin upload segment... Oct 27 15:28:23.560802119 (45m40.910109405s) [seg_pos={0 0}, upload=668]: Done begin upload segment. Oct 27 15:28:23.560839660 (45m40.910146947s) [noise=true , address=145.79.227.20:4437 , piece_id=2EGSCSNCQJFXYT54PWEM33SP3RUYICBKKE7IDRQNMZ2ONPM6I5UA, seg_pos={0 0}, upload=668]: Uploading piece... Oct 27 15:28:23.560885057 (45m40.910192353s) [noise=true , address=145.79.227.19:4079 , piece_id=QXNYHJS2SBTG2QRH7AKW7NZ6SJG7MFO5DHXCMBZHGT6PJTOZ526Q, seg_pos={0 0}, upload=668]: Uploading piece... Oct 27 15:28:23.561607883 (45m40.910915169s) [noise=true , address=145.79.227.26:4653 , piece_id=MXREE4NCFOGYEHVQNFHZYD5DLD4PHBJS64JYSLTXXJZG7AC3LH6Q, seg_pos={0 0}, upload=668]: Uploading piece... Oct 27 15:28:23.561670783 (45m40.910978069s) [noise=true , address=77.90.31.10:4518 , piece_id=76MPB5O5XOR54TOXYCUNK2RBQMWYTWADCBAZV2KQYJ5K7XXDVFMQ, seg_pos={0 0}, upload=668]: Uploading piece... Oct 27 15:28:23.561966716 (45m40.911274002s) [noise=true , address=145.79.227.10:4401 , piece_id=SZNSLFKN44YVSUHRXZTF6BBVPYQODYVEBWQF3JHP6ZDA2A723WAA, seg_pos={0 0}, upload=668]: Uploading piece... Oct 27 15:28:23.562189701 (45m40.911496987s) [noise=true , address=145.79.227.4:4186 , piece_id=6APC73J3JHGVX3ZNCN3UL5DLHXRYORPQT6AYPFPX6ZQLQIRF5TNA, seg_pos={0 0}, upload=668]: Uploading piece... Oct 27 15:28:23.562277909 (45m40.911585195s) [noise=true , address=145.79.227.7:4294 , piece_id=QFLUZYLIGHO5E3ABEA5LFJPHAKDOF3J55Q4BRF5NXVKDNEHV5DBA, seg_pos={0 0}, upload=668]: Uploading piece... Oct 27 15:28:23.562572600 (45m40.911879887s) [noise=true , address=145.79.227.5:4222 , piece_id=B7W6PCNLLAPHXL7XOSJXEOVX6XZGNFDJWXR7MVYDGCONQCN55HYA, seg_pos={0 0}, upload=668]: Uploading piece... Oct 27 15:28:23.562586527 (45m40.911893813s) [noise=true , address=145.79.227.9:4366 , piece_id=5XLRE4OGUCQGZA5UXVNTNTFAMAGGQNHRR4MX23L3362K2DZLDREQ, seg_pos={0 0}, upload=668]: Uploading piece... Oct 27 15:28:23.562888732 (45m40.912196018s) [noise=true , address=145.79.227.21:4475 , piece_id=VDOFRCJKDQLXV4CBM4MA3UGK2PZMRJ3BW6H7PKJRZMZXW56Y5G4A, seg_pos={0 0}, upload=668]: Uploading piece... Oct 27 15:28:23.562953846 (45m40.912261123s) [noise=true , address=145.79.227.17:4971 , piece_id=NK55UPWQTKDWAAMPQTJPPOMZXH5VQGQ3QHJKRNMIRPFLFKQBZZIA, seg_pos={0 0}, upload=668]: Uploading piece... Oct 27 15:28:23.563182882 (45m40.912490158s) [noise=true , address=77.90.31.18:4807 , piece_id=OPDBUFUCTPZN2HH2SORHJL5L4EX2VWXATEFEIIEN5QTBQQFRGSZA, seg_pos={0 0}, upload=668]: Uploading piece... Oct 27 15:28:23.563335423 (45m40.912642709s) [noise=true , address=145.79.227.24:4582 , piece_id=K7COJQ2UG3CTGF4JVO27PTGWY4E3XGGRQWKC3LW24Y565B52YJCQ, seg_pos={0 0}, upload=668]: Uploading piece... Oct 27 15:28:23.563479036 (45m40.912786323s) [noise=true , address=77.90.31.13:4626 , piece_id=GR3XECVFMZ5E3HDMKK2HLKJREMIOZGWIUI7J3NSK7QGB523G7YJQ, seg_pos={0 0}, upload=668]: Uploading piece... Oct 27 15:28:23.563711468 (45m40.913018754s) [noise=true , address=45.91.116.5:4298 , piece_id=ATXRIPPIX6BAHUXCENJVDJ25K2Z6UH3QYL4MTBWPZKP7LFGHBX6A, seg_pos={0 0}, upload=668]: Uploading piece... Oct 27 15:28:23.564137260 (45m40.913444546s) [noise=true , address=145.79.227.5:4223 , piece_id=FHAUWKUQAHGYR6ZJHKBCXT3ZPBG42MMJSADUGU7CIFUO3JPPNKKA, seg_pos={0 0}, upload=668]: Uploading piece... Oct 27 15:28:27.241369967 (45m44.590677243s) [upload=668]: Next returned nil segment Oct 27 15:28:27.241385446 (45m44.590692732s) [upload=668]: Waiting for error group managing segments... Oct 27 15:28:29.744963349 (45m47.094270636s) [noise=true , address=145.79.227.5:4222 , piece_id=B7W6PCNLLAPHXL7XOSJXEOVX6XZGNFDJWXR7MVYDGCONQCN55HYA, seg_pos={0 0}, upload=668]: Done uploading piece. err: <nil> Oct 27 15:28:29.821202504 (45m47.17050978s) [noise=true , address=145.79.227.21:4475 , piece_id=VDOFRCJKDQLXV4CBM4MA3UGK2PZMRJ3BW6H7PKJRZMZXW56Y5G4A, seg_pos={0 0}, upload=668]: Done uploading piece. err: <nil> Oct 27 15:28:32.126653324 (45m49.47596061s) [noise=true , address=77.90.31.18:4807 , piece_id=OPDBUFUCTPZN2HH2SORHJL5L4EX2VWXATEFEIIEN5QTBQQFRGSZA, seg_pos={0 0}, upload=668]: Done uploading piece. err: <nil> Oct 27 15:28:33.087272849 (45m50.436580125s) [noise=true , address=45.91.116.5:4298 , piece_id=ATXRIPPIX6BAHUXCENJVDJ25K2Z6UH3QYL4MTBWPZKP7LFGHBX6A, seg_pos={0 0}, upload=668]: Done uploading piece. err: <nil> Oct 27 15:28:38.581899084 (45m55.93120637s) [noise=true , address=145.79.227.26:4653 , piece_id=MXREE4NCFOGYEHVQNFHZYD5DLD4PHBJS64JYSLTXXJZG7AC3LH6Q, seg_pos={0 0}, upload=668]: Done uploading piece. err: <nil> Oct 27 15:28:39.053322174 (45m56.40262946s) [noise=true , address=145.79.227.24:4582 , piece_id=K7COJQ2UG3CTGF4JVO27PTGWY4E3XGGRQWKC3LW24Y565B52YJCQ, seg_pos={0 0}, upload=668]: Done uploading piece. err: <nil> Oct 27 15:28:39.717124420 (45m57.066431696s) [noise=true , address=145.79.227.20:4437 , piece_id=2EGSCSNCQJFXYT54PWEM33SP3RUYICBKKE7IDRQNMZ2ONPM6I5UA, seg_pos={0 0}, upload=668]: Done uploading piece. err: <nil> Oct 27 15:28:41.618415745 (45m58.967723031s) [noise=true , address=77.90.31.13:4626 , piece_id=GR3XECVFMZ5E3HDMKK2HLKJREMIOZGWIUI7J3NSK7QGB523G7YJQ, seg_pos={0 0}, upload=668]: Done uploading piece. err: <nil> Oct 27 15:28:42.068020890 (45m59.417328166s) [noise=true , address=145.79.227.5:4223 , piece_id=FHAUWKUQAHGYR6ZJHKBCXT3ZPBG42MMJSADUGU7CIFUO3JPPNKKA, seg_pos={0 0}, upload=668]: Done uploading piece. err: <nil> Oct 27 15:28:42.150893235 (45m59.500200521s) [noise=true , address=77.90.31.10:4518 , piece_id=76MPB5O5XOR54TOXYCUNK2RBQMWYTWADCBAZV2KQYJ5K7XXDVFMQ, seg_pos={0 0}, upload=668]: Done uploading piece. err: <nil> Oct 27 15:28:43.703285619 (46m1.052592905s) [noise=true , address=145.79.227.9:4366 , piece_id=5XLRE4OGUCQGZA5UXVNTNTFAMAGGQNHRR4MX23L3362K2DZLDREQ, seg_pos={0 0}, upload=668]: Done uploading piece. err: <nil> Oct 27 15:28:49.699904003 (46m7.049211289s) [noise=true , address=145.79.227.4:4186 , piece_id=6APC73J3JHGVX3ZNCN3UL5DLHXRYORPQT6AYPFPX6ZQLQIRF5TNA, seg_pos={0 0}, upload=668]: Done uploading piece. err: <nil> Oct 27 15:28:49.842045738 (46m7.191353014s) [noise=true , address=145.79.227.7:4294 , piece_id=QFLUZYLIGHO5E3ABEA5LFJPHAKDOF3J55Q4BRF5NXVKDNEHV5DBA, seg_pos={0 0}, upload=668]: Done uploading piece. err: <nil> Oct 27 15:28:49.842081656 (46m7.191388932s) [seg_pos={0 0}, upload=668]: Segment reached optimal threshold of 13 pieces. Oct 27 15:28:49.842296495 (46m7.191603781s) [noise=true , address=145.79.227.17:4971 , piece_id=NK55UPWQTKDWAAMPQTJPPOMZXH5VQGQ3QHJKRNMIRPFLFKQBZZIA, seg_pos={0 0}, upload=668]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo): protocol: write tcp 188.166.225.170:55424->145.79.227.17:4971: use of closed network connection; write tcp 188.166.225.170:55424->145.79.227.17:4971: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55424->145.79.227.17:4971: use of closed network connection Oct 27 15:28:49.842306875 (46m7.191614161s) [noise=true , address=145.79.227.17:4971 , piece_id=NK55UPWQTKDWAAMPQTJPPOMZXH5VQGQ3QHJKRNMIRPFLFKQBZZIA, seg_pos={0 0}, upload=668]: Long tail cancellation due to optimal threshold Oct 27 15:28:49.842335229 (46m7.191642535s) [noise=true , address=145.79.227.10:4401 , piece_id=SZNSLFKN44YVSUHRXZTF6BBVPYQODYVEBWQF3JHP6ZDA2A723WAA, seg_pos={0 0}, upload=668]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1): protocol: write tcp 188.166.225.170:50900->145.79.227.10:4401: use of closed network connection; write tcp 188.166.225.170:50900->145.79.227.10:4401: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50900->145.79.227.10:4401: use of closed network connection Oct 27 15:28:49.842355578 (46m7.191662874s) [noise=true , address=145.79.227.10:4401 , piece_id=SZNSLFKN44YVSUHRXZTF6BBVPYQODYVEBWQF3JHP6ZDA2A723WAA, seg_pos={0 0}, upload=668]: Long tail cancellation due to optimal threshold Oct 27 15:28:49.842403068 (46m7.191710354s) [noise=true , address=145.79.227.19:4079 , piece_id=QXNYHJS2SBTG2QRH7AKW7NZ6SJG7MFO5DHXCMBZHGT6PJTOZ526Q, seg_pos={0 0}, upload=668]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud): protocol: write tcp 188.166.225.170:37220->145.79.227.19:4079: use of closed network connection; write tcp 188.166.225.170:37220->145.79.227.19:4079: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37220->145.79.227.19:4079: use of closed network connection Oct 27 15:28:49.842411143 (46m7.191718419s) [noise=true , address=145.79.227.19:4079 , piece_id=QXNYHJS2SBTG2QRH7AKW7NZ6SJG7MFO5DHXCMBZHGT6PJTOZ526Q, seg_pos={0 0}, upload=668]: Long tail cancellation due to optimal threshold Oct 27 15:28:49.842428026 (46m7.191735312s) [seg_pos={0 0}, upload=668]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 27 15:28:49.842507998 (46m7.191815284s) [upload=668]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 27 15:28:49.842513478 (46m7.191820764s) [upload=668]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 27 15:28:50.170126807 (46m7.519434093s) [upload=668]: Done uploading segments... Oct 27 15:28:50.170148138 (46m7.519455424s) [upload=668]: Upload finished. err: <nil> Duration: 6.447331653s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 3/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=3) [..............X..............XX................X............] duration: 3.405574452s FAIL (2025-10-27 15:39:02 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.405600993s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 6.051129459s FAIL (2025-10-27 15:38:16 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, Piece ID: 76MPB5O5XOR54TOXYCUNK2RBQMWYTWADCBAZV2KQYJ5K7XXDVFMQ) context deadline exceeded; piecestore: (Node ID: 12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, Piece ID: GR3XECVFMZ5E3HDMKK2HLKJREMIOZGWIUI7J3NSK7QGB523G7YJQ) context deadline exceeded; piecestore: (Node ID: 12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, Piece ID: FHAUWKUQAHGYR6ZJHKBCXT3ZPBG42MMJSADUGU7CIFUO3JPPNKKA) context deadline exceeded; piecestore: (Node ID: 1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV, Piece ID: 2EGSCSNCQJFXYT54PWEM33SP3RUYICBKKE7IDRQNMZ2ONPM6I5UA) context deadline exceeded ======LOGS====== ]: Done starting upload Oct 27 15:28:22.900340512 (45m40.249647808s) [upload=668]: Uploading segments... Oct 27 15:28:22.900564057 (45m40.249871343s) [upload=668]: Got next segment. Inline: false Oct 27 15:28:22.900613051 (45m40.249920337s) [upload=668]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 27 15:28:22.900620896 (45m40.249928182s) [upload=668]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 27 15:28:23.560720563 (45m40.91002784s) [seg_pos={0 0}, upload=668]: Begin upload segment... Oct 27 15:28:23.560802119 (45m40.910109405s) [seg_pos={0 0}, upload=668]: Done begin upload segment. Oct 27 15:28:23.560839660 (45m40.910146947s) [noise=true , address=145.79.227.20:4437 , piece_id=2EGSCSNCQJFXYT54PWEM33SP3RUYICBKKE7IDRQNMZ2ONPM6I5UA, seg_pos={0 0}, upload=668]: Uploading piece... Oct 27 15:28:23.560885057 (45m40.910192353s) [noise=true , address=145.79.227.19:4079 , piece_id=QXNYHJS2SBTG2QRH7AKW7NZ6SJG7MFO5DHXCMBZHGT6PJTOZ526Q, seg_pos={0 0}, upload=668]: Uploading piece... Oct 27 15:28:23.561607883 (45m40.910915169s) [noise=true , address=145.79.227.26:4653 , piece_id=MXREE4NCFOGYEHVQNFHZYD5DLD4PHBJS64JYSLTXXJZG7AC3LH6Q, seg_pos={0 0}, upload=668]: Uploading piece... Oct 27 15:28:23.561670783 (45m40.910978069s) [noise=true , address=77.90.31.10:4518 , piece_id=76MPB5O5XOR54TOXYCUNK2RBQMWYTWADCBAZV2KQYJ5K7XXDVFMQ, seg_pos={0 0}, upload=668]: Uploading piece... Oct 27 15:28:23.561966716 (45m40.911274002s) [noise=true , address=145.79.227.10:4401 , piece_id=SZNSLFKN44YVSUHRXZTF6BBVPYQODYVEBWQF3JHP6ZDA2A723WAA, seg_pos={0 0}, upload=668]: Uploading piece... Oct 27 15:28:23.562189701 (45m40.911496987s) [noise=true , address=145.79.227.4:4186 , piece_id=6APC73J3JHGVX3ZNCN3UL5DLHXRYORPQT6AYPFPX6ZQLQIRF5TNA, seg_pos={0 0}, upload=668]: Uploading piece... Oct 27 15:28:23.562277909 (45m40.911585195s) [noise=true , address=145.79.227.7:4294 , piece_id=QFLUZYLIGHO5E3ABEA5LFJPHAKDOF3J55Q4BRF5NXVKDNEHV5DBA, seg_pos={0 0}, upload=668]: Uploading piece... Oct 27 15:28:23.562572600 (45m40.911879887s) [noise=true , address=145.79.227.5:4222 , piece_id=B7W6PCNLLAPHXL7XOSJXEOVX6XZGNFDJWXR7MVYDGCONQCN55HYA, seg_pos={0 0}, upload=668]: Uploading piece... Oct 27 15:28:23.562586527 (45m40.911893813s) [noise=true , address=145.79.227.9:4366 , piece_id=5XLRE4OGUCQGZA5UXVNTNTFAMAGGQNHRR4MX23L3362K2DZLDREQ, seg_pos={0 0}, upload=668]: Uploading piece... Oct 27 15:28:23.562888732 (45m40.912196018s) [noise=true , address=145.79.227.21:4475 , piece_id=VDOFRCJKDQLXV4CBM4MA3UGK2PZMRJ3BW6H7PKJRZMZXW56Y5G4A, seg_pos={0 0}, upload=668]: Uploading piece... Oct 27 15:28:23.562953846 (45m40.912261123s) [noise=true , address=145.79.227.17:4971 , piece_id=NK55UPWQTKDWAAMPQTJPPOMZXH5VQGQ3QHJKRNMIRPFLFKQBZZIA, seg_pos={0 0}, upload=668]: Uploading piece... Oct 27 15:28:23.563182882 (45m40.912490158s) [noise=true , address=77.90.31.18:4807 , piece_id=OPDBUFUCTPZN2HH2SORHJL5L4EX2VWXATEFEIIEN5QTBQQFRGSZA, seg_pos={0 0}, upload=668]: Uploading piece... Oct 27 15:28:23.563335423 (45m40.912642709s) [noise=true , address=145.79.227.24:4582 , piece_id=K7COJQ2UG3CTGF4JVO27PTGWY4E3XGGRQWKC3LW24Y565B52YJCQ, seg_pos={0 0}, upload=668]: Uploading piece... Oct 27 15:28:23.563479036 (45m40.912786323s) [noise=true , address=77.90.31.13:4626 , piece_id=GR3XECVFMZ5E3HDMKK2HLKJREMIOZGWIUI7J3NSK7QGB523G7YJQ, seg_pos={0 0}, upload=668]: Uploading piece... Oct 27 15:28:23.563711468 (45m40.913018754s) [noise=true , address=45.91.116.5:4298 , piece_id=ATXRIPPIX6BAHUXCENJVDJ25K2Z6UH3QYL4MTBWPZKP7LFGHBX6A, seg_pos={0 0}, upload=668]: Uploading piece... Oct 27 15:28:23.564137260 (45m40.913444546s) [noise=true , address=145.79.227.5:4223 , piece_id=FHAUWKUQAHGYR6ZJHKBCXT3ZPBG42MMJSADUGU7CIFUO3JPPNKKA, seg_pos={0 0}, upload=668]: Uploading piece... Oct 27 15:28:27.241369967 (45m44.590677243s) [upload=668]: Next returned nil segment Oct 27 15:28:27.241385446 (45m44.590692732s) [upload=668]: Waiting for error group managing segments... Oct 27 15:28:29.744963349 (45m47.094270636s) [noise=true , address=145.79.227.5:4222 , piece_id=B7W6PCNLLAPHXL7XOSJXEOVX6XZGNFDJWXR7MVYDGCONQCN55HYA, seg_pos={0 0}, upload=668]: Done uploading piece. err: <nil> Oct 27 15:28:29.821202504 (45m47.17050978s) [noise=true , address=145.79.227.21:4475 , piece_id=VDOFRCJKDQLXV4CBM4MA3UGK2PZMRJ3BW6H7PKJRZMZXW56Y5G4A, seg_pos={0 0}, upload=668]: Done uploading piece. err: <nil> Oct 27 15:28:32.126653324 (45m49.47596061s) [noise=true , address=77.90.31.18:4807 , piece_id=OPDBUFUCTPZN2HH2SORHJL5L4EX2VWXATEFEIIEN5QTBQQFRGSZA, seg_pos={0 0}, upload=668]: Done uploading piece. err: <nil> Oct 27 15:28:33.087272849 (45m50.436580125s) [noise=true , address=45.91.116.5:4298 , piece_id=ATXRIPPIX6BAHUXCENJVDJ25K2Z6UH3QYL4MTBWPZKP7LFGHBX6A, seg_pos={0 0}, upload=668]: Done uploading piece. err: <nil> Oct 27 15:28:38.581899084 (45m55.93120637s) [noise=true , address=145.79.227.26:4653 , piece_id=MXREE4NCFOGYEHVQNFHZYD5DLD4PHBJS64JYSLTXXJZG7AC3LH6Q, seg_pos={0 0}, upload=668]: Done uploading piece. err: <nil> Oct 27 15:28:39.053322174 (45m56.40262946s) [noise=true , address=145.79.227.24:4582 , piece_id=K7COJQ2UG3CTGF4JVO27PTGWY4E3XGGRQWKC3LW24Y565B52YJCQ, seg_pos={0 0}, upload=668]: Done uploading piece. err: <nil> Oct 27 15:28:39.717124420 (45m57.066431696s) [noise=true , address=145.79.227.20:4437 , piece_id=2EGSCSNCQJFXYT54PWEM33SP3RUYICBKKE7IDRQNMZ2ONPM6I5UA, seg_pos={0 0}, upload=668]: Done uploading piece. err: <nil> Oct 27 15:28:41.618415745 (45m58.967723031s) [noise=true , address=77.90.31.13:4626 , piece_id=GR3XECVFMZ5E3HDMKK2HLKJREMIOZGWIUI7J3NSK7QGB523G7YJQ, seg_pos={0 0}, upload=668]: Done uploading piece. err: <nil> Oct 27 15:28:42.068020890 (45m59.417328166s) [noise=true , address=145.79.227.5:4223 , piece_id=FHAUWKUQAHGYR6ZJHKBCXT3ZPBG42MMJSADUGU7CIFUO3JPPNKKA, seg_pos={0 0}, upload=668]: Done uploading piece. err: <nil> Oct 27 15:28:42.150893235 (45m59.500200521s) [noise=true , address=77.90.31.10:4518 , piece_id=76MPB5O5XOR54TOXYCUNK2RBQMWYTWADCBAZV2KQYJ5K7XXDVFMQ, seg_pos={0 0}, upload=668]: Done uploading piece. err: <nil> Oct 27 15:28:43.703285619 (46m1.052592905s) [noise=true , address=145.79.227.9:4366 , piece_id=5XLRE4OGUCQGZA5UXVNTNTFAMAGGQNHRR4MX23L3362K2DZLDREQ, seg_pos={0 0}, upload=668]: Done uploading piece. err: <nil> Oct 27 15:28:49.699904003 (46m7.049211289s) [noise=true , address=145.79.227.4:4186 , piece_id=6APC73J3JHGVX3ZNCN3UL5DLHXRYORPQT6AYPFPX6ZQLQIRF5TNA, seg_pos={0 0}, upload=668]: Done uploading piece. err: <nil> Oct 27 15:28:49.842045738 (46m7.191353014s) [noise=true , address=145.79.227.7:4294 , piece_id=QFLUZYLIGHO5E3ABEA5LFJPHAKDOF3J55Q4BRF5NXVKDNEHV5DBA, seg_pos={0 0}, upload=668]: Done uploading piece. err: <nil> Oct 27 15:28:49.842081656 (46m7.191388932s) [seg_pos={0 0}, upload=668]: Segment reached optimal threshold of 13 pieces. Oct 27 15:28:49.842296495 (46m7.191603781s) [noise=true , address=145.79.227.17:4971 , piece_id=NK55UPWQTKDWAAMPQTJPPOMZXH5VQGQ3QHJKRNMIRPFLFKQBZZIA, seg_pos={0 0}, upload=668]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo): protocol: write tcp 188.166.225.170:55424->145.79.227.17:4971: use of closed network connection; write tcp 188.166.225.170:55424->145.79.227.17:4971: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55424->145.79.227.17:4971: use of closed network connection Oct 27 15:28:49.842306875 (46m7.191614161s) [noise=true , address=145.79.227.17:4971 , piece_id=NK55UPWQTKDWAAMPQTJPPOMZXH5VQGQ3QHJKRNMIRPFLFKQBZZIA, seg_pos={0 0}, upload=668]: Long tail cancellation due to optimal threshold Oct 27 15:28:49.842335229 (46m7.191642535s) [noise=true , address=145.79.227.10:4401 , piece_id=SZNSLFKN44YVSUHRXZTF6BBVPYQODYVEBWQF3JHP6ZDA2A723WAA, seg_pos={0 0}, upload=668]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1): protocol: write tcp 188.166.225.170:50900->145.79.227.10:4401: use of closed network connection; write tcp 188.166.225.170:50900->145.79.227.10:4401: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50900->145.79.227.10:4401: use of closed network connection Oct 27 15:28:49.842355578 (46m7.191662874s) [noise=true , address=145.79.227.10:4401 , piece_id=SZNSLFKN44YVSUHRXZTF6BBVPYQODYVEBWQF3JHP6ZDA2A723WAA, seg_pos={0 0}, upload=668]: Long tail cancellation due to optimal threshold Oct 27 15:28:49.842403068 (46m7.191710354s) [noise=true , address=145.79.227.19:4079 , piece_id=QXNYHJS2SBTG2QRH7AKW7NZ6SJG7MFO5DHXCMBZHGT6PJTOZ526Q, seg_pos={0 0}, upload=668]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud): protocol: write tcp 188.166.225.170:37220->145.79.227.19:4079: use of closed network connection; write tcp 188.166.225.170:37220->145.79.227.19:4079: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37220->145.79.227.19:4079: use of closed network connection Oct 27 15:28:49.842411143 (46m7.191718419s) [noise=true , address=145.79.227.19:4079 , piece_id=QXNYHJS2SBTG2QRH7AKW7NZ6SJG7MFO5DHXCMBZHGT6PJTOZ526Q, seg_pos={0 0}, upload=668]: Long tail cancellation due to optimal threshold Oct 27 15:28:49.842428026 (46m7.191735312s) [seg_pos={0 0}, upload=668]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 27 15:28:49.842507998 (46m7.191815284s) [upload=668]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 27 15:28:49.842513478 (46m7.191820764s) [upload=668]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 27 15:28:50.170126807 (46m7.519434093s) [upload=668]: Done uploading segments... Oct 27 15:28:50.170148138 (46m7.519455424s) [upload=668]: Upload finished. err: <nil> Duration: 6.051158255s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 6.205448195s FAIL (2025-10-27 15:38:14 UTC) orbiter warning: context deadline exceeded Duration: 6.205475457s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 3/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=3) [...............X..............XX................X...........] duration: 3.52860479s FAIL (2025-10-27 15:38:05 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.528650787s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 3/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=3) [................X..............XX................X..........] duration: 3.522297638s FAIL (2025-10-27 15:37:43 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.522326193s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 6.084359818s FAIL (2025-10-27 15:37:35 UTC) orbiter warning: context deadline exceeded Duration: 6.084393242s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 6.116577498s FAIL (2025-10-27 15:37:23 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, Piece ID: 76MPB5O5XOR54TOXYCUNK2RBQMWYTWADCBAZV2KQYJ5K7XXDVFMQ) context deadline exceeded; piecestore: (Node ID: 12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, Piece ID: GR3XECVFMZ5E3HDMKK2HLKJREMIOZGWIUI7J3NSK7QGB523G7YJQ) context deadline exceeded; piecestore: (Node ID: 12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, Piece ID: FHAUWKUQAHGYR6ZJHKBCXT3ZPBG42MMJSADUGU7CIFUO3JPPNKKA) context deadline exceeded; piecestore: (Node ID: 1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV, Piece ID: 2EGSCSNCQJFXYT54PWEM33SP3RUYICBKKE7IDRQNMZ2ONPM6I5UA) context deadline exceeded ======LOGS====== ]: Done starting upload Oct 27 15:28:22.900340512 (45m40.249647808s) [upload=668]: Uploading segments... Oct 27 15:28:22.900564057 (45m40.249871343s) [upload=668]: Got next segment. Inline: false Oct 27 15:28:22.900613051 (45m40.249920337s) [upload=668]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 27 15:28:22.900620896 (45m40.249928182s) [upload=668]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 27 15:28:23.560720563 (45m40.91002784s) [seg_pos={0 0}, upload=668]: Begin upload segment... Oct 27 15:28:23.560802119 (45m40.910109405s) [seg_pos={0 0}, upload=668]: Done begin upload segment. Oct 27 15:28:23.560839660 (45m40.910146947s) [noise=true , address=145.79.227.20:4437 , piece_id=2EGSCSNCQJFXYT54PWEM33SP3RUYICBKKE7IDRQNMZ2ONPM6I5UA, seg_pos={0 0}, upload=668]: Uploading piece... Oct 27 15:28:23.560885057 (45m40.910192353s) [noise=true , address=145.79.227.19:4079 , piece_id=QXNYHJS2SBTG2QRH7AKW7NZ6SJG7MFO5DHXCMBZHGT6PJTOZ526Q, seg_pos={0 0}, upload=668]: Uploading piece... Oct 27 15:28:23.561607883 (45m40.910915169s) [noise=true , address=145.79.227.26:4653 , piece_id=MXREE4NCFOGYEHVQNFHZYD5DLD4PHBJS64JYSLTXXJZG7AC3LH6Q, seg_pos={0 0}, upload=668]: Uploading piece... Oct 27 15:28:23.561670783 (45m40.910978069s) [noise=true , address=77.90.31.10:4518 , piece_id=76MPB5O5XOR54TOXYCUNK2RBQMWYTWADCBAZV2KQYJ5K7XXDVFMQ, seg_pos={0 0}, upload=668]: Uploading piece... Oct 27 15:28:23.561966716 (45m40.911274002s) [noise=true , address=145.79.227.10:4401 , piece_id=SZNSLFKN44YVSUHRXZTF6BBVPYQODYVEBWQF3JHP6ZDA2A723WAA, seg_pos={0 0}, upload=668]: Uploading piece... Oct 27 15:28:23.562189701 (45m40.911496987s) [noise=true , address=145.79.227.4:4186 , piece_id=6APC73J3JHGVX3ZNCN3UL5DLHXRYORPQT6AYPFPX6ZQLQIRF5TNA, seg_pos={0 0}, upload=668]: Uploading piece... Oct 27 15:28:23.562277909 (45m40.911585195s) [noise=true , address=145.79.227.7:4294 , piece_id=QFLUZYLIGHO5E3ABEA5LFJPHAKDOF3J55Q4BRF5NXVKDNEHV5DBA, seg_pos={0 0}, upload=668]: Uploading piece... Oct 27 15:28:23.562572600 (45m40.911879887s) [noise=true , address=145.79.227.5:4222 , piece_id=B7W6PCNLLAPHXL7XOSJXEOVX6XZGNFDJWXR7MVYDGCONQCN55HYA, seg_pos={0 0}, upload=668]: Uploading piece... Oct 27 15:28:23.562586527 (45m40.911893813s) [noise=true , address=145.79.227.9:4366 , piece_id=5XLRE4OGUCQGZA5UXVNTNTFAMAGGQNHRR4MX23L3362K2DZLDREQ, seg_pos={0 0}, upload=668]: Uploading piece... Oct 27 15:28:23.562888732 (45m40.912196018s) [noise=true , address=145.79.227.21:4475 , piece_id=VDOFRCJKDQLXV4CBM4MA3UGK2PZMRJ3BW6H7PKJRZMZXW56Y5G4A, seg_pos={0 0}, upload=668]: Uploading piece... Oct 27 15:28:23.562953846 (45m40.912261123s) [noise=true , address=145.79.227.17:4971 , piece_id=NK55UPWQTKDWAAMPQTJPPOMZXH5VQGQ3QHJKRNMIRPFLFKQBZZIA, seg_pos={0 0}, upload=668]: Uploading piece... Oct 27 15:28:23.563182882 (45m40.912490158s) [noise=true , address=77.90.31.18:4807 , piece_id=OPDBUFUCTPZN2HH2SORHJL5L4EX2VWXATEFEIIEN5QTBQQFRGSZA, seg_pos={0 0}, upload=668]: Uploading piece... Oct 27 15:28:23.563335423 (45m40.912642709s) [noise=true , address=145.79.227.24:4582 , piece_id=K7COJQ2UG3CTGF4JVO27PTGWY4E3XGGRQWKC3LW24Y565B52YJCQ, seg_pos={0 0}, upload=668]: Uploading piece... Oct 27 15:28:23.563479036 (45m40.912786323s) [noise=true , address=77.90.31.13:4626 , piece_id=GR3XECVFMZ5E3HDMKK2HLKJREMIOZGWIUI7J3NSK7QGB523G7YJQ, seg_pos={0 0}, upload=668]: Uploading piece... Oct 27 15:28:23.563711468 (45m40.913018754s) [noise=true , address=45.91.116.5:4298 , piece_id=ATXRIPPIX6BAHUXCENJVDJ25K2Z6UH3QYL4MTBWPZKP7LFGHBX6A, seg_pos={0 0}, upload=668]: Uploading piece... Oct 27 15:28:23.564137260 (45m40.913444546s) [noise=true , address=145.79.227.5:4223 , piece_id=FHAUWKUQAHGYR6ZJHKBCXT3ZPBG42MMJSADUGU7CIFUO3JPPNKKA, seg_pos={0 0}, upload=668]: Uploading piece... Oct 27 15:28:27.241369967 (45m44.590677243s) [upload=668]: Next returned nil segment Oct 27 15:28:27.241385446 (45m44.590692732s) [upload=668]: Waiting for error group managing segments... Oct 27 15:28:29.744963349 (45m47.094270636s) [noise=true , address=145.79.227.5:4222 , piece_id=B7W6PCNLLAPHXL7XOSJXEOVX6XZGNFDJWXR7MVYDGCONQCN55HYA, seg_pos={0 0}, upload=668]: Done uploading piece. err: <nil> Oct 27 15:28:29.821202504 (45m47.17050978s) [noise=true , address=145.79.227.21:4475 , piece_id=VDOFRCJKDQLXV4CBM4MA3UGK2PZMRJ3BW6H7PKJRZMZXW56Y5G4A, seg_pos={0 0}, upload=668]: Done uploading piece. err: <nil> Oct 27 15:28:32.126653324 (45m49.47596061s) [noise=true , address=77.90.31.18:4807 , piece_id=OPDBUFUCTPZN2HH2SORHJL5L4EX2VWXATEFEIIEN5QTBQQFRGSZA, seg_pos={0 0}, upload=668]: Done uploading piece. err: <nil> Oct 27 15:28:33.087272849 (45m50.436580125s) [noise=true , address=45.91.116.5:4298 , piece_id=ATXRIPPIX6BAHUXCENJVDJ25K2Z6UH3QYL4MTBWPZKP7LFGHBX6A, seg_pos={0 0}, upload=668]: Done uploading piece. err: <nil> Oct 27 15:28:38.581899084 (45m55.93120637s) [noise=true , address=145.79.227.26:4653 , piece_id=MXREE4NCFOGYEHVQNFHZYD5DLD4PHBJS64JYSLTXXJZG7AC3LH6Q, seg_pos={0 0}, upload=668]: Done uploading piece. err: <nil> Oct 27 15:28:39.053322174 (45m56.40262946s) [noise=true , address=145.79.227.24:4582 , piece_id=K7COJQ2UG3CTGF4JVO27PTGWY4E3XGGRQWKC3LW24Y565B52YJCQ, seg_pos={0 0}, upload=668]: Done uploading piece. err: <nil> Oct 27 15:28:39.717124420 (45m57.066431696s) [noise=true , address=145.79.227.20:4437 , piece_id=2EGSCSNCQJFXYT54PWEM33SP3RUYICBKKE7IDRQNMZ2ONPM6I5UA, seg_pos={0 0}, upload=668]: Done uploading piece. err: <nil> Oct 27 15:28:41.618415745 (45m58.967723031s) [noise=true , address=77.90.31.13:4626 , piece_id=GR3XECVFMZ5E3HDMKK2HLKJREMIOZGWIUI7J3NSK7QGB523G7YJQ, seg_pos={0 0}, upload=668]: Done uploading piece. err: <nil> Oct 27 15:28:42.068020890 (45m59.417328166s) [noise=true , address=145.79.227.5:4223 , piece_id=FHAUWKUQAHGYR6ZJHKBCXT3ZPBG42MMJSADUGU7CIFUO3JPPNKKA, seg_pos={0 0}, upload=668]: Done uploading piece. err: <nil> Oct 27 15:28:42.150893235 (45m59.500200521s) [noise=true , address=77.90.31.10:4518 , piece_id=76MPB5O5XOR54TOXYCUNK2RBQMWYTWADCBAZV2KQYJ5K7XXDVFMQ, seg_pos={0 0}, upload=668]: Done uploading piece. err: <nil> Oct 27 15:28:43.703285619 (46m1.052592905s) [noise=true , address=145.79.227.9:4366 , piece_id=5XLRE4OGUCQGZA5UXVNTNTFAMAGGQNHRR4MX23L3362K2DZLDREQ, seg_pos={0 0}, upload=668]: Done uploading piece. err: <nil> Oct 27 15:28:49.699904003 (46m7.049211289s) [noise=true , address=145.79.227.4:4186 , piece_id=6APC73J3JHGVX3ZNCN3UL5DLHXRYORPQT6AYPFPX6ZQLQIRF5TNA, seg_pos={0 0}, upload=668]: Done uploading piece. err: <nil> Oct 27 15:28:49.842045738 (46m7.191353014s) [noise=true , address=145.79.227.7:4294 , piece_id=QFLUZYLIGHO5E3ABEA5LFJPHAKDOF3J55Q4BRF5NXVKDNEHV5DBA, seg_pos={0 0}, upload=668]: Done uploading piece. err: <nil> Oct 27 15:28:49.842081656 (46m7.191388932s) [seg_pos={0 0}, upload=668]: Segment reached optimal threshold of 13 pieces. Oct 27 15:28:49.842296495 (46m7.191603781s) [noise=true , address=145.79.227.17:4971 , piece_id=NK55UPWQTKDWAAMPQTJPPOMZXH5VQGQ3QHJKRNMIRPFLFKQBZZIA, seg_pos={0 0}, upload=668]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo): protocol: write tcp 188.166.225.170:55424->145.79.227.17:4971: use of closed network connection; write tcp 188.166.225.170:55424->145.79.227.17:4971: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55424->145.79.227.17:4971: use of closed network connection Oct 27 15:28:49.842306875 (46m7.191614161s) [noise=true , address=145.79.227.17:4971 , piece_id=NK55UPWQTKDWAAMPQTJPPOMZXH5VQGQ3QHJKRNMIRPFLFKQBZZIA, seg_pos={0 0}, upload=668]: Long tail cancellation due to optimal threshold Oct 27 15:28:49.842335229 (46m7.191642535s) [noise=true , address=145.79.227.10:4401 , piece_id=SZNSLFKN44YVSUHRXZTF6BBVPYQODYVEBWQF3JHP6ZDA2A723WAA, seg_pos={0 0}, upload=668]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1): protocol: write tcp 188.166.225.170:50900->145.79.227.10:4401: use of closed network connection; write tcp 188.166.225.170:50900->145.79.227.10:4401: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50900->145.79.227.10:4401: use of closed network connection Oct 27 15:28:49.842355578 (46m7.191662874s) [noise=true , address=145.79.227.10:4401 , piece_id=SZNSLFKN44YVSUHRXZTF6BBVPYQODYVEBWQF3JHP6ZDA2A723WAA, seg_pos={0 0}, upload=668]: Long tail cancellation due to optimal threshold Oct 27 15:28:49.842403068 (46m7.191710354s) [noise=true , address=145.79.227.19:4079 , piece_id=QXNYHJS2SBTG2QRH7AKW7NZ6SJG7MFO5DHXCMBZHGT6PJTOZ526Q, seg_pos={0 0}, upload=668]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud): protocol: write tcp 188.166.225.170:37220->145.79.227.19:4079: use of closed network connection; write tcp 188.166.225.170:37220->145.79.227.19:4079: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37220->145.79.227.19:4079: use of closed network connection Oct 27 15:28:49.842411143 (46m7.191718419s) [noise=true , address=145.79.227.19:4079 , piece_id=QXNYHJS2SBTG2QRH7AKW7NZ6SJG7MFO5DHXCMBZHGT6PJTOZ526Q, seg_pos={0 0}, upload=668]: Long tail cancellation due to optimal threshold Oct 27 15:28:49.842428026 (46m7.191735312s) [seg_pos={0 0}, upload=668]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 27 15:28:49.842507998 (46m7.191815284s) [upload=668]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 27 15:28:49.842513478 (46m7.191820764s) [upload=668]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 27 15:28:50.170126807 (46m7.519434093s) [upload=668]: Done uploading segments... Oct 27 15:28:50.170148138 (46m7.519455424s) [upload=668]: Upload finished. err: <nil> Duration: 6.116606093s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 6.040150239s FAIL (2025-10-27 15:36:45 UTC) orbiter warning: context deadline exceeded Duration: 6.040172162s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 3/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=3) [.................X..............XX................X.........] duration: 3.409982987s FAIL (2025-10-27 15:36:33 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.410050254s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 6.188540468s FAIL (2025-10-27 15:36:27 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, Piece ID: 76MPB5O5XOR54TOXYCUNK2RBQMWYTWADCBAZV2KQYJ5K7XXDVFMQ) context deadline exceeded; piecestore: (Node ID: 12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, Piece ID: GR3XECVFMZ5E3HDMKK2HLKJREMIOZGWIUI7J3NSK7QGB523G7YJQ) context deadline exceeded; piecestore: (Node ID: 12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, Piece ID: FHAUWKUQAHGYR6ZJHKBCXT3ZPBG42MMJSADUGU7CIFUO3JPPNKKA) context deadline exceeded; piecestore: (Node ID: 1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV, Piece ID: 2EGSCSNCQJFXYT54PWEM33SP3RUYICBKKE7IDRQNMZ2ONPM6I5UA) context deadline exceeded ======LOGS====== ]: Done starting upload Oct 27 15:28:22.900340512 (45m40.249647808s) [upload=668]: Uploading segments... Oct 27 15:28:22.900564057 (45m40.249871343s) [upload=668]: Got next segment. Inline: false Oct 27 15:28:22.900613051 (45m40.249920337s) [upload=668]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 27 15:28:22.900620896 (45m40.249928182s) [upload=668]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 27 15:28:23.560720563 (45m40.91002784s) [seg_pos={0 0}, upload=668]: Begin upload segment... Oct 27 15:28:23.560802119 (45m40.910109405s) [seg_pos={0 0}, upload=668]: Done begin upload segment. Oct 27 15:28:23.560839660 (45m40.910146947s) [noise=true , address=145.79.227.20:4437 , piece_id=2EGSCSNCQJFXYT54PWEM33SP3RUYICBKKE7IDRQNMZ2ONPM6I5UA, seg_pos={0 0}, upload=668]: Uploading piece... Oct 27 15:28:23.560885057 (45m40.910192353s) [noise=true , address=145.79.227.19:4079 , piece_id=QXNYHJS2SBTG2QRH7AKW7NZ6SJG7MFO5DHXCMBZHGT6PJTOZ526Q, seg_pos={0 0}, upload=668]: Uploading piece... Oct 27 15:28:23.561607883 (45m40.910915169s) [noise=true , address=145.79.227.26:4653 , piece_id=MXREE4NCFOGYEHVQNFHZYD5DLD4PHBJS64JYSLTXXJZG7AC3LH6Q, seg_pos={0 0}, upload=668]: Uploading piece... Oct 27 15:28:23.561670783 (45m40.910978069s) [noise=true , address=77.90.31.10:4518 , piece_id=76MPB5O5XOR54TOXYCUNK2RBQMWYTWADCBAZV2KQYJ5K7XXDVFMQ, seg_pos={0 0}, upload=668]: Uploading piece... Oct 27 15:28:23.561966716 (45m40.911274002s) [noise=true , address=145.79.227.10:4401 , piece_id=SZNSLFKN44YVSUHRXZTF6BBVPYQODYVEBWQF3JHP6ZDA2A723WAA, seg_pos={0 0}, upload=668]: Uploading piece... Oct 27 15:28:23.562189701 (45m40.911496987s) [noise=true , address=145.79.227.4:4186 , piece_id=6APC73J3JHGVX3ZNCN3UL5DLHXRYORPQT6AYPFPX6ZQLQIRF5TNA, seg_pos={0 0}, upload=668]: Uploading piece... Oct 27 15:28:23.562277909 (45m40.911585195s) [noise=true , address=145.79.227.7:4294 , piece_id=QFLUZYLIGHO5E3ABEA5LFJPHAKDOF3J55Q4BRF5NXVKDNEHV5DBA, seg_pos={0 0}, upload=668]: Uploading piece... Oct 27 15:28:23.562572600 (45m40.911879887s) [noise=true , address=145.79.227.5:4222 , piece_id=B7W6PCNLLAPHXL7XOSJXEOVX6XZGNFDJWXR7MVYDGCONQCN55HYA, seg_pos={0 0}, upload=668]: Uploading piece... Oct 27 15:28:23.562586527 (45m40.911893813s) [noise=true , address=145.79.227.9:4366 , piece_id=5XLRE4OGUCQGZA5UXVNTNTFAMAGGQNHRR4MX23L3362K2DZLDREQ, seg_pos={0 0}, upload=668]: Uploading piece... Oct 27 15:28:23.562888732 (45m40.912196018s) [noise=true , address=145.79.227.21:4475 , piece_id=VDOFRCJKDQLXV4CBM4MA3UGK2PZMRJ3BW6H7PKJRZMZXW56Y5G4A, seg_pos={0 0}, upload=668]: Uploading piece... Oct 27 15:28:23.562953846 (45m40.912261123s) [noise=true , address=145.79.227.17:4971 , piece_id=NK55UPWQTKDWAAMPQTJPPOMZXH5VQGQ3QHJKRNMIRPFLFKQBZZIA, seg_pos={0 0}, upload=668]: Uploading piece... Oct 27 15:28:23.563182882 (45m40.912490158s) [noise=true , address=77.90.31.18:4807 , piece_id=OPDBUFUCTPZN2HH2SORHJL5L4EX2VWXATEFEIIEN5QTBQQFRGSZA, seg_pos={0 0}, upload=668]: Uploading piece... Oct 27 15:28:23.563335423 (45m40.912642709s) [noise=true , address=145.79.227.24:4582 , piece_id=K7COJQ2UG3CTGF4JVO27PTGWY4E3XGGRQWKC3LW24Y565B52YJCQ, seg_pos={0 0}, upload=668]: Uploading piece... Oct 27 15:28:23.563479036 (45m40.912786323s) [noise=true , address=77.90.31.13:4626 , piece_id=GR3XECVFMZ5E3HDMKK2HLKJREMIOZGWIUI7J3NSK7QGB523G7YJQ, seg_pos={0 0}, upload=668]: Uploading piece... Oct 27 15:28:23.563711468 (45m40.913018754s) [noise=true , address=45.91.116.5:4298 , piece_id=ATXRIPPIX6BAHUXCENJVDJ25K2Z6UH3QYL4MTBWPZKP7LFGHBX6A, seg_pos={0 0}, upload=668]: Uploading piece... Oct 27 15:28:23.564137260 (45m40.913444546s) [noise=true , address=145.79.227.5:4223 , piece_id=FHAUWKUQAHGYR6ZJHKBCXT3ZPBG42MMJSADUGU7CIFUO3JPPNKKA, seg_pos={0 0}, upload=668]: Uploading piece... Oct 27 15:28:27.241369967 (45m44.590677243s) [upload=668]: Next returned nil segment Oct 27 15:28:27.241385446 (45m44.590692732s) [upload=668]: Waiting for error group managing segments... Oct 27 15:28:29.744963349 (45m47.094270636s) [noise=true , address=145.79.227.5:4222 , piece_id=B7W6PCNLLAPHXL7XOSJXEOVX6XZGNFDJWXR7MVYDGCONQCN55HYA, seg_pos={0 0}, upload=668]: Done uploading piece. err: <nil> Oct 27 15:28:29.821202504 (45m47.17050978s) [noise=true , address=145.79.227.21:4475 , piece_id=VDOFRCJKDQLXV4CBM4MA3UGK2PZMRJ3BW6H7PKJRZMZXW56Y5G4A, seg_pos={0 0}, upload=668]: Done uploading piece. err: <nil> Oct 27 15:28:32.126653324 (45m49.47596061s) [noise=true , address=77.90.31.18:4807 , piece_id=OPDBUFUCTPZN2HH2SORHJL5L4EX2VWXATEFEIIEN5QTBQQFRGSZA, seg_pos={0 0}, upload=668]: Done uploading piece. err: <nil> Oct 27 15:28:33.087272849 (45m50.436580125s) [noise=true , address=45.91.116.5:4298 , piece_id=ATXRIPPIX6BAHUXCENJVDJ25K2Z6UH3QYL4MTBWPZKP7LFGHBX6A, seg_pos={0 0}, upload=668]: Done uploading piece. err: <nil> Oct 27 15:28:38.581899084 (45m55.93120637s) [noise=true , address=145.79.227.26:4653 , piece_id=MXREE4NCFOGYEHVQNFHZYD5DLD4PHBJS64JYSLTXXJZG7AC3LH6Q, seg_pos={0 0}, upload=668]: Done uploading piece. err: <nil> Oct 27 15:28:39.053322174 (45m56.40262946s) [noise=true , address=145.79.227.24:4582 , piece_id=K7COJQ2UG3CTGF4JVO27PTGWY4E3XGGRQWKC3LW24Y565B52YJCQ, seg_pos={0 0}, upload=668]: Done uploading piece. err: <nil> Oct 27 15:28:39.717124420 (45m57.066431696s) [noise=true , address=145.79.227.20:4437 , piece_id=2EGSCSNCQJFXYT54PWEM33SP3RUYICBKKE7IDRQNMZ2ONPM6I5UA, seg_pos={0 0}, upload=668]: Done uploading piece. err: <nil> Oct 27 15:28:41.618415745 (45m58.967723031s) [noise=true , address=77.90.31.13:4626 , piece_id=GR3XECVFMZ5E3HDMKK2HLKJREMIOZGWIUI7J3NSK7QGB523G7YJQ, seg_pos={0 0}, upload=668]: Done uploading piece. err: <nil> Oct 27 15:28:42.068020890 (45m59.417328166s) [noise=true , address=145.79.227.5:4223 , piece_id=FHAUWKUQAHGYR6ZJHKBCXT3ZPBG42MMJSADUGU7CIFUO3JPPNKKA, seg_pos={0 0}, upload=668]: Done uploading piece. err: <nil> Oct 27 15:28:42.150893235 (45m59.500200521s) [noise=true , address=77.90.31.10:4518 , piece_id=76MPB5O5XOR54TOXYCUNK2RBQMWYTWADCBAZV2KQYJ5K7XXDVFMQ, seg_pos={0 0}, upload=668]: Done uploading piece. err: <nil> Oct 27 15:28:43.703285619 (46m1.052592905s) [noise=true , address=145.79.227.9:4366 , piece_id=5XLRE4OGUCQGZA5UXVNTNTFAMAGGQNHRR4MX23L3362K2DZLDREQ, seg_pos={0 0}, upload=668]: Done uploading piece. err: <nil> Oct 27 15:28:49.699904003 (46m7.049211289s) [noise=true , address=145.79.227.4:4186 , piece_id=6APC73J3JHGVX3ZNCN3UL5DLHXRYORPQT6AYPFPX6ZQLQIRF5TNA, seg_pos={0 0}, upload=668]: Done uploading piece. err: <nil> Oct 27 15:28:49.842045738 (46m7.191353014s) [noise=true , address=145.79.227.7:4294 , piece_id=QFLUZYLIGHO5E3ABEA5LFJPHAKDOF3J55Q4BRF5NXVKDNEHV5DBA, seg_pos={0 0}, upload=668]: Done uploading piece. err: <nil> Oct 27 15:28:49.842081656 (46m7.191388932s) [seg_pos={0 0}, upload=668]: Segment reached optimal threshold of 13 pieces. Oct 27 15:28:49.842296495 (46m7.191603781s) [noise=true , address=145.79.227.17:4971 , piece_id=NK55UPWQTKDWAAMPQTJPPOMZXH5VQGQ3QHJKRNMIRPFLFKQBZZIA, seg_pos={0 0}, upload=668]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo): protocol: write tcp 188.166.225.170:55424->145.79.227.17:4971: use of closed network connection; write tcp 188.166.225.170:55424->145.79.227.17:4971: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55424->145.79.227.17:4971: use of closed network connection Oct 27 15:28:49.842306875 (46m7.191614161s) [noise=true , address=145.79.227.17:4971 , piece_id=NK55UPWQTKDWAAMPQTJPPOMZXH5VQGQ3QHJKRNMIRPFLFKQBZZIA, seg_pos={0 0}, upload=668]: Long tail cancellation due to optimal threshold Oct 27 15:28:49.842335229 (46m7.191642535s) [noise=true , address=145.79.227.10:4401 , piece_id=SZNSLFKN44YVSUHRXZTF6BBVPYQODYVEBWQF3JHP6ZDA2A723WAA, seg_pos={0 0}, upload=668]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1): protocol: write tcp 188.166.225.170:50900->145.79.227.10:4401: use of closed network connection; write tcp 188.166.225.170:50900->145.79.227.10:4401: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50900->145.79.227.10:4401: use of closed network connection Oct 27 15:28:49.842355578 (46m7.191662874s) [noise=true , address=145.79.227.10:4401 , piece_id=SZNSLFKN44YVSUHRXZTF6BBVPYQODYVEBWQF3JHP6ZDA2A723WAA, seg_pos={0 0}, upload=668]: Long tail cancellation due to optimal threshold Oct 27 15:28:49.842403068 (46m7.191710354s) [noise=true , address=145.79.227.19:4079 , piece_id=QXNYHJS2SBTG2QRH7AKW7NZ6SJG7MFO5DHXCMBZHGT6PJTOZ526Q, seg_pos={0 0}, upload=668]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud): protocol: write tcp 188.166.225.170:37220->145.79.227.19:4079: use of closed network connection; write tcp 188.166.225.170:37220->145.79.227.19:4079: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37220->145.79.227.19:4079: use of closed network connection Oct 27 15:28:49.842411143 (46m7.191718419s) [noise=true , address=145.79.227.19:4079 , piece_id=QXNYHJS2SBTG2QRH7AKW7NZ6SJG7MFO5DHXCMBZHGT6PJTOZ526Q, seg_pos={0 0}, upload=668]: Long tail cancellation due to optimal threshold Oct 27 15:28:49.842428026 (46m7.191735312s) [seg_pos={0 0}, upload=668]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 27 15:28:49.842507998 (46m7.191815284s) [upload=668]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 27 15:28:49.842513478 (46m7.191820764s) [upload=668]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 27 15:28:50.170126807 (46m7.519434093s) [upload=668]: Done uploading segments... Oct 27 15:28:50.170148138 (46m7.519455424s) [upload=668]: Upload finished. err: <nil> Duration: 6.188569915s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 3/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=3) [..................X..............XX................X........] duration: 3.456848107s FAIL (2025-10-27 15:36:18 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.456879077s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 5.920123613s FAIL (2025-10-27 15:35:39 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, Piece ID: 76MPB5O5XOR54TOXYCUNK2RBQMWYTWADCBAZV2KQYJ5K7XXDVFMQ) context deadline exceeded; piecestore: (Node ID: 12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, Piece ID: GR3XECVFMZ5E3HDMKK2HLKJREMIOZGWIUI7J3NSK7QGB523G7YJQ) context deadline exceeded; piecestore: (Node ID: 12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, Piece ID: FHAUWKUQAHGYR6ZJHKBCXT3ZPBG42MMJSADUGU7CIFUO3JPPNKKA) context deadline exceeded; piecestore: (Node ID: 1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV, Piece ID: 2EGSCSNCQJFXYT54PWEM33SP3RUYICBKKE7IDRQNMZ2ONPM6I5UA) context deadline exceeded ======LOGS====== ]: Done starting upload Oct 27 15:28:22.900340512 (45m40.249647808s) [upload=668]: Uploading segments... Oct 27 15:28:22.900564057 (45m40.249871343s) [upload=668]: Got next segment. Inline: false Oct 27 15:28:22.900613051 (45m40.249920337s) [upload=668]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 27 15:28:22.900620896 (45m40.249928182s) [upload=668]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 27 15:28:23.560720563 (45m40.91002784s) [seg_pos={0 0}, upload=668]: Begin upload segment... Oct 27 15:28:23.560802119 (45m40.910109405s) [seg_pos={0 0}, upload=668]: Done begin upload segment. Oct 27 15:28:23.560839660 (45m40.910146947s) [noise=true , address=145.79.227.20:4437 , piece_id=2EGSCSNCQJFXYT54PWEM33SP3RUYICBKKE7IDRQNMZ2ONPM6I5UA, seg_pos={0 0}, upload=668]: Uploading piece... Oct 27 15:28:23.560885057 (45m40.910192353s) [noise=true , address=145.79.227.19:4079 , piece_id=QXNYHJS2SBTG2QRH7AKW7NZ6SJG7MFO5DHXCMBZHGT6PJTOZ526Q, seg_pos={0 0}, upload=668]: Uploading piece... Oct 27 15:28:23.561607883 (45m40.910915169s) [noise=true , address=145.79.227.26:4653 , piece_id=MXREE4NCFOGYEHVQNFHZYD5DLD4PHBJS64JYSLTXXJZG7AC3LH6Q, seg_pos={0 0}, upload=668]: Uploading piece... Oct 27 15:28:23.561670783 (45m40.910978069s) [noise=true , address=77.90.31.10:4518 , piece_id=76MPB5O5XOR54TOXYCUNK2RBQMWYTWADCBAZV2KQYJ5K7XXDVFMQ, seg_pos={0 0}, upload=668]: Uploading piece... Oct 27 15:28:23.561966716 (45m40.911274002s) [noise=true , address=145.79.227.10:4401 , piece_id=SZNSLFKN44YVSUHRXZTF6BBVPYQODYVEBWQF3JHP6ZDA2A723WAA, seg_pos={0 0}, upload=668]: Uploading piece... Oct 27 15:28:23.562189701 (45m40.911496987s) [noise=true , address=145.79.227.4:4186 , piece_id=6APC73J3JHGVX3ZNCN3UL5DLHXRYORPQT6AYPFPX6ZQLQIRF5TNA, seg_pos={0 0}, upload=668]: Uploading piece... Oct 27 15:28:23.562277909 (45m40.911585195s) [noise=true , address=145.79.227.7:4294 , piece_id=QFLUZYLIGHO5E3ABEA5LFJPHAKDOF3J55Q4BRF5NXVKDNEHV5DBA, seg_pos={0 0}, upload=668]: Uploading piece... Oct 27 15:28:23.562572600 (45m40.911879887s) [noise=true , address=145.79.227.5:4222 , piece_id=B7W6PCNLLAPHXL7XOSJXEOVX6XZGNFDJWXR7MVYDGCONQCN55HYA, seg_pos={0 0}, upload=668]: Uploading piece... Oct 27 15:28:23.562586527 (45m40.911893813s) [noise=true , address=145.79.227.9:4366 , piece_id=5XLRE4OGUCQGZA5UXVNTNTFAMAGGQNHRR4MX23L3362K2DZLDREQ, seg_pos={0 0}, upload=668]: Uploading piece... Oct 27 15:28:23.562888732 (45m40.912196018s) [noise=true , address=145.79.227.21:4475 , piece_id=VDOFRCJKDQLXV4CBM4MA3UGK2PZMRJ3BW6H7PKJRZMZXW56Y5G4A, seg_pos={0 0}, upload=668]: Uploading piece... Oct 27 15:28:23.562953846 (45m40.912261123s) [noise=true , address=145.79.227.17:4971 , piece_id=NK55UPWQTKDWAAMPQTJPPOMZXH5VQGQ3QHJKRNMIRPFLFKQBZZIA, seg_pos={0 0}, upload=668]: Uploading piece... Oct 27 15:28:23.563182882 (45m40.912490158s) [noise=true , address=77.90.31.18:4807 , piece_id=OPDBUFUCTPZN2HH2SORHJL5L4EX2VWXATEFEIIEN5QTBQQFRGSZA, seg_pos={0 0}, upload=668]: Uploading piece... Oct 27 15:28:23.563335423 (45m40.912642709s) [noise=true , address=145.79.227.24:4582 , piece_id=K7COJQ2UG3CTGF4JVO27PTGWY4E3XGGRQWKC3LW24Y565B52YJCQ, seg_pos={0 0}, upload=668]: Uploading piece... Oct 27 15:28:23.563479036 (45m40.912786323s) [noise=true , address=77.90.31.13:4626 , piece_id=GR3XECVFMZ5E3HDMKK2HLKJREMIOZGWIUI7J3NSK7QGB523G7YJQ, seg_pos={0 0}, upload=668]: Uploading piece... Oct 27 15:28:23.563711468 (45m40.913018754s) [noise=true , address=45.91.116.5:4298 , piece_id=ATXRIPPIX6BAHUXCENJVDJ25K2Z6UH3QYL4MTBWPZKP7LFGHBX6A, seg_pos={0 0}, upload=668]: Uploading piece... Oct 27 15:28:23.564137260 (45m40.913444546s) [noise=true , address=145.79.227.5:4223 , piece_id=FHAUWKUQAHGYR6ZJHKBCXT3ZPBG42MMJSADUGU7CIFUO3JPPNKKA, seg_pos={0 0}, upload=668]: Uploading piece... Oct 27 15:28:27.241369967 (45m44.590677243s) [upload=668]: Next returned nil segment Oct 27 15:28:27.241385446 (45m44.590692732s) [upload=668]: Waiting for error group managing segments... Oct 27 15:28:29.744963349 (45m47.094270636s) [noise=true , address=145.79.227.5:4222 , piece_id=B7W6PCNLLAPHXL7XOSJXEOVX6XZGNFDJWXR7MVYDGCONQCN55HYA, seg_pos={0 0}, upload=668]: Done uploading piece. err: <nil> Oct 27 15:28:29.821202504 (45m47.17050978s) [noise=true , address=145.79.227.21:4475 , piece_id=VDOFRCJKDQLXV4CBM4MA3UGK2PZMRJ3BW6H7PKJRZMZXW56Y5G4A, seg_pos={0 0}, upload=668]: Done uploading piece. err: <nil> Oct 27 15:28:32.126653324 (45m49.47596061s) [noise=true , address=77.90.31.18:4807 , piece_id=OPDBUFUCTPZN2HH2SORHJL5L4EX2VWXATEFEIIEN5QTBQQFRGSZA, seg_pos={0 0}, upload=668]: Done uploading piece. err: <nil> Oct 27 15:28:33.087272849 (45m50.436580125s) [noise=true , address=45.91.116.5:4298 , piece_id=ATXRIPPIX6BAHUXCENJVDJ25K2Z6UH3QYL4MTBWPZKP7LFGHBX6A, seg_pos={0 0}, upload=668]: Done uploading piece. err: <nil> Oct 27 15:28:38.581899084 (45m55.93120637s) [noise=true , address=145.79.227.26:4653 , piece_id=MXREE4NCFOGYEHVQNFHZYD5DLD4PHBJS64JYSLTXXJZG7AC3LH6Q, seg_pos={0 0}, upload=668]: Done uploading piece. err: <nil> Oct 27 15:28:39.053322174 (45m56.40262946s) [noise=true , address=145.79.227.24:4582 , piece_id=K7COJQ2UG3CTGF4JVO27PTGWY4E3XGGRQWKC3LW24Y565B52YJCQ, seg_pos={0 0}, upload=668]: Done uploading piece. err: <nil> Oct 27 15:28:39.717124420 (45m57.066431696s) [noise=true , address=145.79.227.20:4437 , piece_id=2EGSCSNCQJFXYT54PWEM33SP3RUYICBKKE7IDRQNMZ2ONPM6I5UA, seg_pos={0 0}, upload=668]: Done uploading piece. err: <nil> Oct 27 15:28:41.618415745 (45m58.967723031s) [noise=true , address=77.90.31.13:4626 , piece_id=GR3XECVFMZ5E3HDMKK2HLKJREMIOZGWIUI7J3NSK7QGB523G7YJQ, seg_pos={0 0}, upload=668]: Done uploading piece. err: <nil> Oct 27 15:28:42.068020890 (45m59.417328166s) [noise=true , address=145.79.227.5:4223 , piece_id=FHAUWKUQAHGYR6ZJHKBCXT3ZPBG42MMJSADUGU7CIFUO3JPPNKKA, seg_pos={0 0}, upload=668]: Done uploading piece. err: <nil> Oct 27 15:28:42.150893235 (45m59.500200521s) [noise=true , address=77.90.31.10:4518 , piece_id=76MPB5O5XOR54TOXYCUNK2RBQMWYTWADCBAZV2KQYJ5K7XXDVFMQ, seg_pos={0 0}, upload=668]: Done uploading piece. err: <nil> Oct 27 15:28:43.703285619 (46m1.052592905s) [noise=true , address=145.79.227.9:4366 , piece_id=5XLRE4OGUCQGZA5UXVNTNTFAMAGGQNHRR4MX23L3362K2DZLDREQ, seg_pos={0 0}, upload=668]: Done uploading piece. err: <nil> Oct 27 15:28:49.699904003 (46m7.049211289s) [noise=true , address=145.79.227.4:4186 , piece_id=6APC73J3JHGVX3ZNCN3UL5DLHXRYORPQT6AYPFPX6ZQLQIRF5TNA, seg_pos={0 0}, upload=668]: Done uploading piece. err: <nil> Oct 27 15:28:49.842045738 (46m7.191353014s) [noise=true , address=145.79.227.7:4294 , piece_id=QFLUZYLIGHO5E3ABEA5LFJPHAKDOF3J55Q4BRF5NXVKDNEHV5DBA, seg_pos={0 0}, upload=668]: Done uploading piece. err: <nil> Oct 27 15:28:49.842081656 (46m7.191388932s) [seg_pos={0 0}, upload=668]: Segment reached optimal threshold of 13 pieces. Oct 27 15:28:49.842296495 (46m7.191603781s) [noise=true , address=145.79.227.17:4971 , piece_id=NK55UPWQTKDWAAMPQTJPPOMZXH5VQGQ3QHJKRNMIRPFLFKQBZZIA, seg_pos={0 0}, upload=668]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo): protocol: write tcp 188.166.225.170:55424->145.79.227.17:4971: use of closed network connection; write tcp 188.166.225.170:55424->145.79.227.17:4971: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55424->145.79.227.17:4971: use of closed network connection Oct 27 15:28:49.842306875 (46m7.191614161s) [noise=true , address=145.79.227.17:4971 , piece_id=NK55UPWQTKDWAAMPQTJPPOMZXH5VQGQ3QHJKRNMIRPFLFKQBZZIA, seg_pos={0 0}, upload=668]: Long tail cancellation due to optimal threshold Oct 27 15:28:49.842335229 (46m7.191642535s) [noise=true , address=145.79.227.10:4401 , piece_id=SZNSLFKN44YVSUHRXZTF6BBVPYQODYVEBWQF3JHP6ZDA2A723WAA, seg_pos={0 0}, upload=668]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1): protocol: write tcp 188.166.225.170:50900->145.79.227.10:4401: use of closed network connection; write tcp 188.166.225.170:50900->145.79.227.10:4401: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50900->145.79.227.10:4401: use of closed network connection Oct 27 15:28:49.842355578 (46m7.191662874s) [noise=true , address=145.79.227.10:4401 , piece_id=SZNSLFKN44YVSUHRXZTF6BBVPYQODYVEBWQF3JHP6ZDA2A723WAA, seg_pos={0 0}, upload=668]: Long tail cancellation due to optimal threshold Oct 27 15:28:49.842403068 (46m7.191710354s) [noise=true , address=145.79.227.19:4079 , piece_id=QXNYHJS2SBTG2QRH7AKW7NZ6SJG7MFO5DHXCMBZHGT6PJTOZ526Q, seg_pos={0 0}, upload=668]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud): protocol: write tcp 188.166.225.170:37220->145.79.227.19:4079: use of closed network connection; write tcp 188.166.225.170:37220->145.79.227.19:4079: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37220->145.79.227.19:4079: use of closed network connection Oct 27 15:28:49.842411143 (46m7.191718419s) [noise=true , address=145.79.227.19:4079 , piece_id=QXNYHJS2SBTG2QRH7AKW7NZ6SJG7MFO5DHXCMBZHGT6PJTOZ526Q, seg_pos={0 0}, upload=668]: Long tail cancellation due to optimal threshold Oct 27 15:28:49.842428026 (46m7.191735312s) [seg_pos={0 0}, upload=668]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 27 15:28:49.842507998 (46m7.191815284s) [upload=668]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 27 15:28:49.842513478 (46m7.191820764s) [upload=668]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 27 15:28:50.170126807 (46m7.519434093s) [upload=668]: Done uploading segments... Oct 27 15:28:50.170148138 (46m7.519455424s) [upload=668]: Upload finished. err: <nil> Duration: 5.920151877s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 6.215298421s FAIL (2025-10-27 15:35:33 UTC) orbiter warning: context deadline exceeded Duration: 6.215337065s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 3/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=3) [...................X..............XX................X.......] duration: 3.407483209s FAIL (2025-10-27 15:35:28 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.407505542s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 6.074747495s FAIL (2025-10-27 15:35:22 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, Piece ID: 76MPB5O5XOR54TOXYCUNK2RBQMWYTWADCBAZV2KQYJ5K7XXDVFMQ) context deadline exceeded; piecestore: (Node ID: 12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, Piece ID: GR3XECVFMZ5E3HDMKK2HLKJREMIOZGWIUI7J3NSK7QGB523G7YJQ) context deadline exceeded; piecestore: (Node ID: 12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, Piece ID: FHAUWKUQAHGYR6ZJHKBCXT3ZPBG42MMJSADUGU7CIFUO3JPPNKKA) context deadline exceeded; piecestore: (Node ID: 1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV, Piece ID: 2EGSCSNCQJFXYT54PWEM33SP3RUYICBKKE7IDRQNMZ2ONPM6I5UA) context deadline exceeded ======LOGS====== ]: Done starting upload Oct 27 15:28:22.900340512 (45m40.249647808s) [upload=668]: Uploading segments... Oct 27 15:28:22.900564057 (45m40.249871343s) [upload=668]: Got next segment. Inline: false Oct 27 15:28:22.900613051 (45m40.249920337s) [upload=668]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 27 15:28:22.900620896 (45m40.249928182s) [upload=668]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 27 15:28:23.560720563 (45m40.91002784s) [seg_pos={0 0}, upload=668]: Begin upload segment... Oct 27 15:28:23.560802119 (45m40.910109405s) [seg_pos={0 0}, upload=668]: Done begin upload segment. Oct 27 15:28:23.560839660 (45m40.910146947s) [noise=true , address=145.79.227.20:4437 , piece_id=2EGSCSNCQJFXYT54PWEM33SP3RUYICBKKE7IDRQNMZ2ONPM6I5UA, seg_pos={0 0}, upload=668]: Uploading piece... Oct 27 15:28:23.560885057 (45m40.910192353s) [noise=true , address=145.79.227.19:4079 , piece_id=QXNYHJS2SBTG2QRH7AKW7NZ6SJG7MFO5DHXCMBZHGT6PJTOZ526Q, seg_pos={0 0}, upload=668]: Uploading piece... Oct 27 15:28:23.561607883 (45m40.910915169s) [noise=true , address=145.79.227.26:4653 , piece_id=MXREE4NCFOGYEHVQNFHZYD5DLD4PHBJS64JYSLTXXJZG7AC3LH6Q, seg_pos={0 0}, upload=668]: Uploading piece... Oct 27 15:28:23.561670783 (45m40.910978069s) [noise=true , address=77.90.31.10:4518 , piece_id=76MPB5O5XOR54TOXYCUNK2RBQMWYTWADCBAZV2KQYJ5K7XXDVFMQ, seg_pos={0 0}, upload=668]: Uploading piece... Oct 27 15:28:23.561966716 (45m40.911274002s) [noise=true , address=145.79.227.10:4401 , piece_id=SZNSLFKN44YVSUHRXZTF6BBVPYQODYVEBWQF3JHP6ZDA2A723WAA, seg_pos={0 0}, upload=668]: Uploading piece... Oct 27 15:28:23.562189701 (45m40.911496987s) [noise=true , address=145.79.227.4:4186 , piece_id=6APC73J3JHGVX3ZNCN3UL5DLHXRYORPQT6AYPFPX6ZQLQIRF5TNA, seg_pos={0 0}, upload=668]: Uploading piece... Oct 27 15:28:23.562277909 (45m40.911585195s) [noise=true , address=145.79.227.7:4294 , piece_id=QFLUZYLIGHO5E3ABEA5LFJPHAKDOF3J55Q4BRF5NXVKDNEHV5DBA, seg_pos={0 0}, upload=668]: Uploading piece... Oct 27 15:28:23.562572600 (45m40.911879887s) [noise=true , address=145.79.227.5:4222 , piece_id=B7W6PCNLLAPHXL7XOSJXEOVX6XZGNFDJWXR7MVYDGCONQCN55HYA, seg_pos={0 0}, upload=668]: Uploading piece... Oct 27 15:28:23.562586527 (45m40.911893813s) [noise=true , address=145.79.227.9:4366 , piece_id=5XLRE4OGUCQGZA5UXVNTNTFAMAGGQNHRR4MX23L3362K2DZLDREQ, seg_pos={0 0}, upload=668]: Uploading piece... Oct 27 15:28:23.562888732 (45m40.912196018s) [noise=true , address=145.79.227.21:4475 , piece_id=VDOFRCJKDQLXV4CBM4MA3UGK2PZMRJ3BW6H7PKJRZMZXW56Y5G4A, seg_pos={0 0}, upload=668]: Uploading piece... Oct 27 15:28:23.562953846 (45m40.912261123s) [noise=true , address=145.79.227.17:4971 , piece_id=NK55UPWQTKDWAAMPQTJPPOMZXH5VQGQ3QHJKRNMIRPFLFKQBZZIA, seg_pos={0 0}, upload=668]: Uploading piece... Oct 27 15:28:23.563182882 (45m40.912490158s) [noise=true , address=77.90.31.18:4807 , piece_id=OPDBUFUCTPZN2HH2SORHJL5L4EX2VWXATEFEIIEN5QTBQQFRGSZA, seg_pos={0 0}, upload=668]: Uploading piece... Oct 27 15:28:23.563335423 (45m40.912642709s) [noise=true , address=145.79.227.24:4582 , piece_id=K7COJQ2UG3CTGF4JVO27PTGWY4E3XGGRQWKC3LW24Y565B52YJCQ, seg_pos={0 0}, upload=668]: Uploading piece... Oct 27 15:28:23.563479036 (45m40.912786323s) [noise=true , address=77.90.31.13:4626 , piece_id=GR3XECVFMZ5E3HDMKK2HLKJREMIOZGWIUI7J3NSK7QGB523G7YJQ, seg_pos={0 0}, upload=668]: Uploading piece... Oct 27 15:28:23.563711468 (45m40.913018754s) [noise=true , address=45.91.116.5:4298 , piece_id=ATXRIPPIX6BAHUXCENJVDJ25K2Z6UH3QYL4MTBWPZKP7LFGHBX6A, seg_pos={0 0}, upload=668]: Uploading piece... Oct 27 15:28:23.564137260 (45m40.913444546s) [noise=true , address=145.79.227.5:4223 , piece_id=FHAUWKUQAHGYR6ZJHKBCXT3ZPBG42MMJSADUGU7CIFUO3JPPNKKA, seg_pos={0 0}, upload=668]: Uploading piece... Oct 27 15:28:27.241369967 (45m44.590677243s) [upload=668]: Next returned nil segment Oct 27 15:28:27.241385446 (45m44.590692732s) [upload=668]: Waiting for error group managing segments... Oct 27 15:28:29.744963349 (45m47.094270636s) [noise=true , address=145.79.227.5:4222 , piece_id=B7W6PCNLLAPHXL7XOSJXEOVX6XZGNFDJWXR7MVYDGCONQCN55HYA, seg_pos={0 0}, upload=668]: Done uploading piece. err: <nil> Oct 27 15:28:29.821202504 (45m47.17050978s) [noise=true , address=145.79.227.21:4475 , piece_id=VDOFRCJKDQLXV4CBM4MA3UGK2PZMRJ3BW6H7PKJRZMZXW56Y5G4A, seg_pos={0 0}, upload=668]: Done uploading piece. err: <nil> Oct 27 15:28:32.126653324 (45m49.47596061s) [noise=true , address=77.90.31.18:4807 , piece_id=OPDBUFUCTPZN2HH2SORHJL5L4EX2VWXATEFEIIEN5QTBQQFRGSZA, seg_pos={0 0}, upload=668]: Done uploading piece. err: <nil> Oct 27 15:28:33.087272849 (45m50.436580125s) [noise=true , address=45.91.116.5:4298 , piece_id=ATXRIPPIX6BAHUXCENJVDJ25K2Z6UH3QYL4MTBWPZKP7LFGHBX6A, seg_pos={0 0}, upload=668]: Done uploading piece. err: <nil> Oct 27 15:28:38.581899084 (45m55.93120637s) [noise=true , address=145.79.227.26:4653 , piece_id=MXREE4NCFOGYEHVQNFHZYD5DLD4PHBJS64JYSLTXXJZG7AC3LH6Q, seg_pos={0 0}, upload=668]: Done uploading piece. err: <nil> Oct 27 15:28:39.053322174 (45m56.40262946s) [noise=true , address=145.79.227.24:4582 , piece_id=K7COJQ2UG3CTGF4JVO27PTGWY4E3XGGRQWKC3LW24Y565B52YJCQ, seg_pos={0 0}, upload=668]: Done uploading piece. err: <nil> Oct 27 15:28:39.717124420 (45m57.066431696s) [noise=true , address=145.79.227.20:4437 , piece_id=2EGSCSNCQJFXYT54PWEM33SP3RUYICBKKE7IDRQNMZ2ONPM6I5UA, seg_pos={0 0}, upload=668]: Done uploading piece. err: <nil> Oct 27 15:28:41.618415745 (45m58.967723031s) [noise=true , address=77.90.31.13:4626 , piece_id=GR3XECVFMZ5E3HDMKK2HLKJREMIOZGWIUI7J3NSK7QGB523G7YJQ, seg_pos={0 0}, upload=668]: Done uploading piece. err: <nil> Oct 27 15:28:42.068020890 (45m59.417328166s) [noise=true , address=145.79.227.5:4223 , piece_id=FHAUWKUQAHGYR6ZJHKBCXT3ZPBG42MMJSADUGU7CIFUO3JPPNKKA, seg_pos={0 0}, upload=668]: Done uploading piece. err: <nil> Oct 27 15:28:42.150893235 (45m59.500200521s) [noise=true , address=77.90.31.10:4518 , piece_id=76MPB5O5XOR54TOXYCUNK2RBQMWYTWADCBAZV2KQYJ5K7XXDVFMQ, seg_pos={0 0}, upload=668]: Done uploading piece. err: <nil> Oct 27 15:28:43.703285619 (46m1.052592905s) [noise=true , address=145.79.227.9:4366 , piece_id=5XLRE4OGUCQGZA5UXVNTNTFAMAGGQNHRR4MX23L3362K2DZLDREQ, seg_pos={0 0}, upload=668]: Done uploading piece. err: <nil> Oct 27 15:28:49.699904003 (46m7.049211289s) [noise=true , address=145.79.227.4:4186 , piece_id=6APC73J3JHGVX3ZNCN3UL5DLHXRYORPQT6AYPFPX6ZQLQIRF5TNA, seg_pos={0 0}, upload=668]: Done uploading piece. err: <nil> Oct 27 15:28:49.842045738 (46m7.191353014s) [noise=true , address=145.79.227.7:4294 , piece_id=QFLUZYLIGHO5E3ABEA5LFJPHAKDOF3J55Q4BRF5NXVKDNEHV5DBA, seg_pos={0 0}, upload=668]: Done uploading piece. err: <nil> Oct 27 15:28:49.842081656 (46m7.191388932s) [seg_pos={0 0}, upload=668]: Segment reached optimal threshold of 13 pieces. Oct 27 15:28:49.842296495 (46m7.191603781s) [noise=true , address=145.79.227.17:4971 , piece_id=NK55UPWQTKDWAAMPQTJPPOMZXH5VQGQ3QHJKRNMIRPFLFKQBZZIA, seg_pos={0 0}, upload=668]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo): protocol: write tcp 188.166.225.170:55424->145.79.227.17:4971: use of closed network connection; write tcp 188.166.225.170:55424->145.79.227.17:4971: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55424->145.79.227.17:4971: use of closed network connection Oct 27 15:28:49.842306875 (46m7.191614161s) [noise=true , address=145.79.227.17:4971 , piece_id=NK55UPWQTKDWAAMPQTJPPOMZXH5VQGQ3QHJKRNMIRPFLFKQBZZIA, seg_pos={0 0}, upload=668]: Long tail cancellation due to optimal threshold Oct 27 15:28:49.842335229 (46m7.191642535s) [noise=true , address=145.79.227.10:4401 , piece_id=SZNSLFKN44YVSUHRXZTF6BBVPYQODYVEBWQF3JHP6ZDA2A723WAA, seg_pos={0 0}, upload=668]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1): protocol: write tcp 188.166.225.170:50900->145.79.227.10:4401: use of closed network connection; write tcp 188.166.225.170:50900->145.79.227.10:4401: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50900->145.79.227.10:4401: use of closed network connection Oct 27 15:28:49.842355578 (46m7.191662874s) [noise=true , address=145.79.227.10:4401 , piece_id=SZNSLFKN44YVSUHRXZTF6BBVPYQODYVEBWQF3JHP6ZDA2A723WAA, seg_pos={0 0}, upload=668]: Long tail cancellation due to optimal threshold Oct 27 15:28:49.842403068 (46m7.191710354s) [noise=true , address=145.79.227.19:4079 , piece_id=QXNYHJS2SBTG2QRH7AKW7NZ6SJG7MFO5DHXCMBZHGT6PJTOZ526Q, seg_pos={0 0}, upload=668]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud): protocol: write tcp 188.166.225.170:37220->145.79.227.19:4079: use of closed network connection; write tcp 188.166.225.170:37220->145.79.227.19:4079: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37220->145.79.227.19:4079: use of closed network connection Oct 27 15:28:49.842411143 (46m7.191718419s) [noise=true , address=145.79.227.19:4079 , piece_id=QXNYHJS2SBTG2QRH7AKW7NZ6SJG7MFO5DHXCMBZHGT6PJTOZ526Q, seg_pos={0 0}, upload=668]: Long tail cancellation due to optimal threshold Oct 27 15:28:49.842428026 (46m7.191735312s) [seg_pos={0 0}, upload=668]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 27 15:28:49.842507998 (46m7.191815284s) [upload=668]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 27 15:28:49.842513478 (46m7.191820764s) [upload=668]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 27 15:28:50.170126807 (46m7.519434093s) [upload=668]: Done uploading segments... Oct 27 15:28:50.170148138 (46m7.519455424s) [upload=668]: Upload finished. err: <nil> Duration: 6.074779165s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 6.115185331s FAIL (2025-10-27 15:34:49 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, Piece ID: 76MPB5O5XOR54TOXYCUNK2RBQMWYTWADCBAZV2KQYJ5K7XXDVFMQ) context deadline exceeded; piecestore: (Node ID: 12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, Piece ID: GR3XECVFMZ5E3HDMKK2HLKJREMIOZGWIUI7J3NSK7QGB523G7YJQ) context deadline exceeded; piecestore: (Node ID: 12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, Piece ID: FHAUWKUQAHGYR6ZJHKBCXT3ZPBG42MMJSADUGU7CIFUO3JPPNKKA) context deadline exceeded; piecestore: (Node ID: 1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV, Piece ID: 2EGSCSNCQJFXYT54PWEM33SP3RUYICBKKE7IDRQNMZ2ONPM6I5UA) context deadline exceeded ======LOGS====== ]: Done starting upload Oct 27 15:28:22.900340512 (45m40.249647808s) [upload=668]: Uploading segments... Oct 27 15:28:22.900564057 (45m40.249871343s) [upload=668]: Got next segment. Inline: false Oct 27 15:28:22.900613051 (45m40.249920337s) [upload=668]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 27 15:28:22.900620896 (45m40.249928182s) [upload=668]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 27 15:28:23.560720563 (45m40.91002784s) [seg_pos={0 0}, upload=668]: Begin upload segment... Oct 27 15:28:23.560802119 (45m40.910109405s) [seg_pos={0 0}, upload=668]: Done begin upload segment. Oct 27 15:28:23.560839660 (45m40.910146947s) [noise=true , address=145.79.227.20:4437 , piece_id=2EGSCSNCQJFXYT54PWEM33SP3RUYICBKKE7IDRQNMZ2ONPM6I5UA, seg_pos={0 0}, upload=668]: Uploading piece... Oct 27 15:28:23.560885057 (45m40.910192353s) [noise=true , address=145.79.227.19:4079 , piece_id=QXNYHJS2SBTG2QRH7AKW7NZ6SJG7MFO5DHXCMBZHGT6PJTOZ526Q, seg_pos={0 0}, upload=668]: Uploading piece... Oct 27 15:28:23.561607883 (45m40.910915169s) [noise=true , address=145.79.227.26:4653 , piece_id=MXREE4NCFOGYEHVQNFHZYD5DLD4PHBJS64JYSLTXXJZG7AC3LH6Q, seg_pos={0 0}, upload=668]: Uploading piece... Oct 27 15:28:23.561670783 (45m40.910978069s) [noise=true , address=77.90.31.10:4518 , piece_id=76MPB5O5XOR54TOXYCUNK2RBQMWYTWADCBAZV2KQYJ5K7XXDVFMQ, seg_pos={0 0}, upload=668]: Uploading piece... Oct 27 15:28:23.561966716 (45m40.911274002s) [noise=true , address=145.79.227.10:4401 , piece_id=SZNSLFKN44YVSUHRXZTF6BBVPYQODYVEBWQF3JHP6ZDA2A723WAA, seg_pos={0 0}, upload=668]: Uploading piece... Oct 27 15:28:23.562189701 (45m40.911496987s) [noise=true , address=145.79.227.4:4186 , piece_id=6APC73J3JHGVX3ZNCN3UL5DLHXRYORPQT6AYPFPX6ZQLQIRF5TNA, seg_pos={0 0}, upload=668]: Uploading piece... Oct 27 15:28:23.562277909 (45m40.911585195s) [noise=true , address=145.79.227.7:4294 , piece_id=QFLUZYLIGHO5E3ABEA5LFJPHAKDOF3J55Q4BRF5NXVKDNEHV5DBA, seg_pos={0 0}, upload=668]: Uploading piece... Oct 27 15:28:23.562572600 (45m40.911879887s) [noise=true , address=145.79.227.5:4222 , piece_id=B7W6PCNLLAPHXL7XOSJXEOVX6XZGNFDJWXR7MVYDGCONQCN55HYA, seg_pos={0 0}, upload=668]: Uploading piece... Oct 27 15:28:23.562586527 (45m40.911893813s) [noise=true , address=145.79.227.9:4366 , piece_id=5XLRE4OGUCQGZA5UXVNTNTFAMAGGQNHRR4MX23L3362K2DZLDREQ, seg_pos={0 0}, upload=668]: Uploading piece... Oct 27 15:28:23.562888732 (45m40.912196018s) [noise=true , address=145.79.227.21:4475 , piece_id=VDOFRCJKDQLXV4CBM4MA3UGK2PZMRJ3BW6H7PKJRZMZXW56Y5G4A, seg_pos={0 0}, upload=668]: Uploading piece... Oct 27 15:28:23.562953846 (45m40.912261123s) [noise=true , address=145.79.227.17:4971 , piece_id=NK55UPWQTKDWAAMPQTJPPOMZXH5VQGQ3QHJKRNMIRPFLFKQBZZIA, seg_pos={0 0}, upload=668]: Uploading piece... Oct 27 15:28:23.563182882 (45m40.912490158s) [noise=true , address=77.90.31.18:4807 , piece_id=OPDBUFUCTPZN2HH2SORHJL5L4EX2VWXATEFEIIEN5QTBQQFRGSZA, seg_pos={0 0}, upload=668]: Uploading piece... Oct 27 15:28:23.563335423 (45m40.912642709s) [noise=true , address=145.79.227.24:4582 , piece_id=K7COJQ2UG3CTGF4JVO27PTGWY4E3XGGRQWKC3LW24Y565B52YJCQ, seg_pos={0 0}, upload=668]: Uploading piece... Oct 27 15:28:23.563479036 (45m40.912786323s) [noise=true , address=77.90.31.13:4626 , piece_id=GR3XECVFMZ5E3HDMKK2HLKJREMIOZGWIUI7J3NSK7QGB523G7YJQ, seg_pos={0 0}, upload=668]: Uploading piece... Oct 27 15:28:23.563711468 (45m40.913018754s) [noise=true , address=45.91.116.5:4298 , piece_id=ATXRIPPIX6BAHUXCENJVDJ25K2Z6UH3QYL4MTBWPZKP7LFGHBX6A, seg_pos={0 0}, upload=668]: Uploading piece... Oct 27 15:28:23.564137260 (45m40.913444546s) [noise=true , address=145.79.227.5:4223 , piece_id=FHAUWKUQAHGYR6ZJHKBCXT3ZPBG42MMJSADUGU7CIFUO3JPPNKKA, seg_pos={0 0}, upload=668]: Uploading piece... Oct 27 15:28:27.241369967 (45m44.590677243s) [upload=668]: Next returned nil segment Oct 27 15:28:27.241385446 (45m44.590692732s) [upload=668]: Waiting for error group managing segments... Oct 27 15:28:29.744963349 (45m47.094270636s) [noise=true , address=145.79.227.5:4222 , piece_id=B7W6PCNLLAPHXL7XOSJXEOVX6XZGNFDJWXR7MVYDGCONQCN55HYA, seg_pos={0 0}, upload=668]: Done uploading piece. err: <nil> Oct 27 15:28:29.821202504 (45m47.17050978s) [noise=true , address=145.79.227.21:4475 , piece_id=VDOFRCJKDQLXV4CBM4MA3UGK2PZMRJ3BW6H7PKJRZMZXW56Y5G4A, seg_pos={0 0}, upload=668]: Done uploading piece. err: <nil> Oct 27 15:28:32.126653324 (45m49.47596061s) [noise=true , address=77.90.31.18:4807 , piece_id=OPDBUFUCTPZN2HH2SORHJL5L4EX2VWXATEFEIIEN5QTBQQFRGSZA, seg_pos={0 0}, upload=668]: Done uploading piece. err: <nil> Oct 27 15:28:33.087272849 (45m50.436580125s) [noise=true , address=45.91.116.5:4298 , piece_id=ATXRIPPIX6BAHUXCENJVDJ25K2Z6UH3QYL4MTBWPZKP7LFGHBX6A, seg_pos={0 0}, upload=668]: Done uploading piece. err: <nil> Oct 27 15:28:38.581899084 (45m55.93120637s) [noise=true , address=145.79.227.26:4653 , piece_id=MXREE4NCFOGYEHVQNFHZYD5DLD4PHBJS64JYSLTXXJZG7AC3LH6Q, seg_pos={0 0}, upload=668]: Done uploading piece. err: <nil> Oct 27 15:28:39.053322174 (45m56.40262946s) [noise=true , address=145.79.227.24:4582 , piece_id=K7COJQ2UG3CTGF4JVO27PTGWY4E3XGGRQWKC3LW24Y565B52YJCQ, seg_pos={0 0}, upload=668]: Done uploading piece. err: <nil> Oct 27 15:28:39.717124420 (45m57.066431696s) [noise=true , address=145.79.227.20:4437 , piece_id=2EGSCSNCQJFXYT54PWEM33SP3RUYICBKKE7IDRQNMZ2ONPM6I5UA, seg_pos={0 0}, upload=668]: Done uploading piece. err: <nil> Oct 27 15:28:41.618415745 (45m58.967723031s) [noise=true , address=77.90.31.13:4626 , piece_id=GR3XECVFMZ5E3HDMKK2HLKJREMIOZGWIUI7J3NSK7QGB523G7YJQ, seg_pos={0 0}, upload=668]: Done uploading piece. err: <nil> Oct 27 15:28:42.068020890 (45m59.417328166s) [noise=true , address=145.79.227.5:4223 , piece_id=FHAUWKUQAHGYR6ZJHKBCXT3ZPBG42MMJSADUGU7CIFUO3JPPNKKA, seg_pos={0 0}, upload=668]: Done uploading piece. err: <nil> Oct 27 15:28:42.150893235 (45m59.500200521s) [noise=true , address=77.90.31.10:4518 , piece_id=76MPB5O5XOR54TOXYCUNK2RBQMWYTWADCBAZV2KQYJ5K7XXDVFMQ, seg_pos={0 0}, upload=668]: Done uploading piece. err: <nil> Oct 27 15:28:43.703285619 (46m1.052592905s) [noise=true , address=145.79.227.9:4366 , piece_id=5XLRE4OGUCQGZA5UXVNTNTFAMAGGQNHRR4MX23L3362K2DZLDREQ, seg_pos={0 0}, upload=668]: Done uploading piece. err: <nil> Oct 27 15:28:49.699904003 (46m7.049211289s) [noise=true , address=145.79.227.4:4186 , piece_id=6APC73J3JHGVX3ZNCN3UL5DLHXRYORPQT6AYPFPX6ZQLQIRF5TNA, seg_pos={0 0}, upload=668]: Done uploading piece. err: <nil> Oct 27 15:28:49.842045738 (46m7.191353014s) [noise=true , address=145.79.227.7:4294 , piece_id=QFLUZYLIGHO5E3ABEA5LFJPHAKDOF3J55Q4BRF5NXVKDNEHV5DBA, seg_pos={0 0}, upload=668]: Done uploading piece. err: <nil> Oct 27 15:28:49.842081656 (46m7.191388932s) [seg_pos={0 0}, upload=668]: Segment reached optimal threshold of 13 pieces. Oct 27 15:28:49.842296495 (46m7.191603781s) [noise=true , address=145.79.227.17:4971 , piece_id=NK55UPWQTKDWAAMPQTJPPOMZXH5VQGQ3QHJKRNMIRPFLFKQBZZIA, seg_pos={0 0}, upload=668]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo): protocol: write tcp 188.166.225.170:55424->145.79.227.17:4971: use of closed network connection; write tcp 188.166.225.170:55424->145.79.227.17:4971: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55424->145.79.227.17:4971: use of closed network connection Oct 27 15:28:49.842306875 (46m7.191614161s) [noise=true , address=145.79.227.17:4971 , piece_id=NK55UPWQTKDWAAMPQTJPPOMZXH5VQGQ3QHJKRNMIRPFLFKQBZZIA, seg_pos={0 0}, upload=668]: Long tail cancellation due to optimal threshold Oct 27 15:28:49.842335229 (46m7.191642535s) [noise=true , address=145.79.227.10:4401 , piece_id=SZNSLFKN44YVSUHRXZTF6BBVPYQODYVEBWQF3JHP6ZDA2A723WAA, seg_pos={0 0}, upload=668]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1): protocol: write tcp 188.166.225.170:50900->145.79.227.10:4401: use of closed network connection; write tcp 188.166.225.170:50900->145.79.227.10:4401: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50900->145.79.227.10:4401: use of closed network connection Oct 27 15:28:49.842355578 (46m7.191662874s) [noise=true , address=145.79.227.10:4401 , piece_id=SZNSLFKN44YVSUHRXZTF6BBVPYQODYVEBWQF3JHP6ZDA2A723WAA, seg_pos={0 0}, upload=668]: Long tail cancellation due to optimal threshold Oct 27 15:28:49.842403068 (46m7.191710354s) [noise=true , address=145.79.227.19:4079 , piece_id=QXNYHJS2SBTG2QRH7AKW7NZ6SJG7MFO5DHXCMBZHGT6PJTOZ526Q, seg_pos={0 0}, upload=668]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud): protocol: write tcp 188.166.225.170:37220->145.79.227.19:4079: use of closed network connection; write tcp 188.166.225.170:37220->145.79.227.19:4079: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37220->145.79.227.19:4079: use of closed network connection Oct 27 15:28:49.842411143 (46m7.191718419s) [noise=true , address=145.79.227.19:4079 , piece_id=QXNYHJS2SBTG2QRH7AKW7NZ6SJG7MFO5DHXCMBZHGT6PJTOZ526Q, seg_pos={0 0}, upload=668]: Long tail cancellation due to optimal threshold Oct 27 15:28:49.842428026 (46m7.191735312s) [seg_pos={0 0}, upload=668]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 27 15:28:49.842507998 (46m7.191815284s) [upload=668]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 27 15:28:49.842513478 (46m7.191820764s) [upload=668]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 27 15:28:50.170126807 (46m7.519434093s) [upload=668]: Done uploading segments... Oct 27 15:28:50.170148138 (46m7.519455424s) [upload=668]: Upload finished. err: <nil> Duration: 6.115222631s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 6.300985074s FAIL (2025-10-27 15:34:27 UTC) orbiter warning: context deadline exceeded Duration: 6.301006895s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 3/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=3) [....................X..............XX................X......] duration: 3.441921847s FAIL (2025-10-27 15:34:18 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.442026325s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 6.372265394s FAIL (2025-10-27 15:34:00 UTC) orbiter warning: context deadline exceeded Duration: 6.372297737s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 5.967864906s FAIL (2025-10-27 15:33:42 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, Piece ID: 76MPB5O5XOR54TOXYCUNK2RBQMWYTWADCBAZV2KQYJ5K7XXDVFMQ) context deadline exceeded; piecestore: (Node ID: 12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, Piece ID: GR3XECVFMZ5E3HDMKK2HLKJREMIOZGWIUI7J3NSK7QGB523G7YJQ) context deadline exceeded; piecestore: (Node ID: 12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, Piece ID: FHAUWKUQAHGYR6ZJHKBCXT3ZPBG42MMJSADUGU7CIFUO3JPPNKKA) context deadline exceeded; piecestore: (Node ID: 1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV, Piece ID: 2EGSCSNCQJFXYT54PWEM33SP3RUYICBKKE7IDRQNMZ2ONPM6I5UA) context deadline exceeded ======LOGS====== ]: Done starting upload Oct 27 15:28:22.900340512 (45m40.249647808s) [upload=668]: Uploading segments... Oct 27 15:28:22.900564057 (45m40.249871343s) [upload=668]: Got next segment. Inline: false Oct 27 15:28:22.900613051 (45m40.249920337s) [upload=668]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 27 15:28:22.900620896 (45m40.249928182s) [upload=668]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 27 15:28:23.560720563 (45m40.91002784s) [seg_pos={0 0}, upload=668]: Begin upload segment... Oct 27 15:28:23.560802119 (45m40.910109405s) [seg_pos={0 0}, upload=668]: Done begin upload segment. Oct 27 15:28:23.560839660 (45m40.910146947s) [noise=true , address=145.79.227.20:4437 , piece_id=2EGSCSNCQJFXYT54PWEM33SP3RUYICBKKE7IDRQNMZ2ONPM6I5UA, seg_pos={0 0}, upload=668]: Uploading piece... Oct 27 15:28:23.560885057 (45m40.910192353s) [noise=true , address=145.79.227.19:4079 , piece_id=QXNYHJS2SBTG2QRH7AKW7NZ6SJG7MFO5DHXCMBZHGT6PJTOZ526Q, seg_pos={0 0}, upload=668]: Uploading piece... Oct 27 15:28:23.561607883 (45m40.910915169s) [noise=true , address=145.79.227.26:4653 , piece_id=MXREE4NCFOGYEHVQNFHZYD5DLD4PHBJS64JYSLTXXJZG7AC3LH6Q, seg_pos={0 0}, upload=668]: Uploading piece... Oct 27 15:28:23.561670783 (45m40.910978069s) [noise=true , address=77.90.31.10:4518 , piece_id=76MPB5O5XOR54TOXYCUNK2RBQMWYTWADCBAZV2KQYJ5K7XXDVFMQ, seg_pos={0 0}, upload=668]: Uploading piece... Oct 27 15:28:23.561966716 (45m40.911274002s) [noise=true , address=145.79.227.10:4401 , piece_id=SZNSLFKN44YVSUHRXZTF6BBVPYQODYVEBWQF3JHP6ZDA2A723WAA, seg_pos={0 0}, upload=668]: Uploading piece... Oct 27 15:28:23.562189701 (45m40.911496987s) [noise=true , address=145.79.227.4:4186 , piece_id=6APC73J3JHGVX3ZNCN3UL5DLHXRYORPQT6AYPFPX6ZQLQIRF5TNA, seg_pos={0 0}, upload=668]: Uploading piece... Oct 27 15:28:23.562277909 (45m40.911585195s) [noise=true , address=145.79.227.7:4294 , piece_id=QFLUZYLIGHO5E3ABEA5LFJPHAKDOF3J55Q4BRF5NXVKDNEHV5DBA, seg_pos={0 0}, upload=668]: Uploading piece... Oct 27 15:28:23.562572600 (45m40.911879887s) [noise=true , address=145.79.227.5:4222 , piece_id=B7W6PCNLLAPHXL7XOSJXEOVX6XZGNFDJWXR7MVYDGCONQCN55HYA, seg_pos={0 0}, upload=668]: Uploading piece... Oct 27 15:28:23.562586527 (45m40.911893813s) [noise=true , address=145.79.227.9:4366 , piece_id=5XLRE4OGUCQGZA5UXVNTNTFAMAGGQNHRR4MX23L3362K2DZLDREQ, seg_pos={0 0}, upload=668]: Uploading piece... Oct 27 15:28:23.562888732 (45m40.912196018s) [noise=true , address=145.79.227.21:4475 , piece_id=VDOFRCJKDQLXV4CBM4MA3UGK2PZMRJ3BW6H7PKJRZMZXW56Y5G4A, seg_pos={0 0}, upload=668]: Uploading piece... Oct 27 15:28:23.562953846 (45m40.912261123s) [noise=true , address=145.79.227.17:4971 , piece_id=NK55UPWQTKDWAAMPQTJPPOMZXH5VQGQ3QHJKRNMIRPFLFKQBZZIA, seg_pos={0 0}, upload=668]: Uploading piece... Oct 27 15:28:23.563182882 (45m40.912490158s) [noise=true , address=77.90.31.18:4807 , piece_id=OPDBUFUCTPZN2HH2SORHJL5L4EX2VWXATEFEIIEN5QTBQQFRGSZA, seg_pos={0 0}, upload=668]: Uploading piece... Oct 27 15:28:23.563335423 (45m40.912642709s) [noise=true , address=145.79.227.24:4582 , piece_id=K7COJQ2UG3CTGF4JVO27PTGWY4E3XGGRQWKC3LW24Y565B52YJCQ, seg_pos={0 0}, upload=668]: Uploading piece... Oct 27 15:28:23.563479036 (45m40.912786323s) [noise=true , address=77.90.31.13:4626 , piece_id=GR3XECVFMZ5E3HDMKK2HLKJREMIOZGWIUI7J3NSK7QGB523G7YJQ, seg_pos={0 0}, upload=668]: Uploading piece... Oct 27 15:28:23.563711468 (45m40.913018754s) [noise=true , address=45.91.116.5:4298 , piece_id=ATXRIPPIX6BAHUXCENJVDJ25K2Z6UH3QYL4MTBWPZKP7LFGHBX6A, seg_pos={0 0}, upload=668]: Uploading piece... Oct 27 15:28:23.564137260 (45m40.913444546s) [noise=true , address=145.79.227.5:4223 , piece_id=FHAUWKUQAHGYR6ZJHKBCXT3ZPBG42MMJSADUGU7CIFUO3JPPNKKA, seg_pos={0 0}, upload=668]: Uploading piece... Oct 27 15:28:27.241369967 (45m44.590677243s) [upload=668]: Next returned nil segment Oct 27 15:28:27.241385446 (45m44.590692732s) [upload=668]: Waiting for error group managing segments... Oct 27 15:28:29.744963349 (45m47.094270636s) [noise=true , address=145.79.227.5:4222 , piece_id=B7W6PCNLLAPHXL7XOSJXEOVX6XZGNFDJWXR7MVYDGCONQCN55HYA, seg_pos={0 0}, upload=668]: Done uploading piece. err: <nil> Oct 27 15:28:29.821202504 (45m47.17050978s) [noise=true , address=145.79.227.21:4475 , piece_id=VDOFRCJKDQLXV4CBM4MA3UGK2PZMRJ3BW6H7PKJRZMZXW56Y5G4A, seg_pos={0 0}, upload=668]: Done uploading piece. err: <nil> Oct 27 15:28:32.126653324 (45m49.47596061s) [noise=true , address=77.90.31.18:4807 , piece_id=OPDBUFUCTPZN2HH2SORHJL5L4EX2VWXATEFEIIEN5QTBQQFRGSZA, seg_pos={0 0}, upload=668]: Done uploading piece. err: <nil> Oct 27 15:28:33.087272849 (45m50.436580125s) [noise=true , address=45.91.116.5:4298 , piece_id=ATXRIPPIX6BAHUXCENJVDJ25K2Z6UH3QYL4MTBWPZKP7LFGHBX6A, seg_pos={0 0}, upload=668]: Done uploading piece. err: <nil> Oct 27 15:28:38.581899084 (45m55.93120637s) [noise=true , address=145.79.227.26:4653 , piece_id=MXREE4NCFOGYEHVQNFHZYD5DLD4PHBJS64JYSLTXXJZG7AC3LH6Q, seg_pos={0 0}, upload=668]: Done uploading piece. err: <nil> Oct 27 15:28:39.053322174 (45m56.40262946s) [noise=true , address=145.79.227.24:4582 , piece_id=K7COJQ2UG3CTGF4JVO27PTGWY4E3XGGRQWKC3LW24Y565B52YJCQ, seg_pos={0 0}, upload=668]: Done uploading piece. err: <nil> Oct 27 15:28:39.717124420 (45m57.066431696s) [noise=true , address=145.79.227.20:4437 , piece_id=2EGSCSNCQJFXYT54PWEM33SP3RUYICBKKE7IDRQNMZ2ONPM6I5UA, seg_pos={0 0}, upload=668]: Done uploading piece. err: <nil> Oct 27 15:28:41.618415745 (45m58.967723031s) [noise=true , address=77.90.31.13:4626 , piece_id=GR3XECVFMZ5E3HDMKK2HLKJREMIOZGWIUI7J3NSK7QGB523G7YJQ, seg_pos={0 0}, upload=668]: Done uploading piece. err: <nil> Oct 27 15:28:42.068020890 (45m59.417328166s) [noise=true , address=145.79.227.5:4223 , piece_id=FHAUWKUQAHGYR6ZJHKBCXT3ZPBG42MMJSADUGU7CIFUO3JPPNKKA, seg_pos={0 0}, upload=668]: Done uploading piece. err: <nil> Oct 27 15:28:42.150893235 (45m59.500200521s) [noise=true , address=77.90.31.10:4518 , piece_id=76MPB5O5XOR54TOXYCUNK2RBQMWYTWADCBAZV2KQYJ5K7XXDVFMQ, seg_pos={0 0}, upload=668]: Done uploading piece. err: <nil> Oct 27 15:28:43.703285619 (46m1.052592905s) [noise=true , address=145.79.227.9:4366 , piece_id=5XLRE4OGUCQGZA5UXVNTNTFAMAGGQNHRR4MX23L3362K2DZLDREQ, seg_pos={0 0}, upload=668]: Done uploading piece. err: <nil> Oct 27 15:28:49.699904003 (46m7.049211289s) [noise=true , address=145.79.227.4:4186 , piece_id=6APC73J3JHGVX3ZNCN3UL5DLHXRYORPQT6AYPFPX6ZQLQIRF5TNA, seg_pos={0 0}, upload=668]: Done uploading piece. err: <nil> Oct 27 15:28:49.842045738 (46m7.191353014s) [noise=true , address=145.79.227.7:4294 , piece_id=QFLUZYLIGHO5E3ABEA5LFJPHAKDOF3J55Q4BRF5NXVKDNEHV5DBA, seg_pos={0 0}, upload=668]: Done uploading piece. err: <nil> Oct 27 15:28:49.842081656 (46m7.191388932s) [seg_pos={0 0}, upload=668]: Segment reached optimal threshold of 13 pieces. Oct 27 15:28:49.842296495 (46m7.191603781s) [noise=true , address=145.79.227.17:4971 , piece_id=NK55UPWQTKDWAAMPQTJPPOMZXH5VQGQ3QHJKRNMIRPFLFKQBZZIA, seg_pos={0 0}, upload=668]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo): protocol: write tcp 188.166.225.170:55424->145.79.227.17:4971: use of closed network connection; write tcp 188.166.225.170:55424->145.79.227.17:4971: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55424->145.79.227.17:4971: use of closed network connection Oct 27 15:28:49.842306875 (46m7.191614161s) [noise=true , address=145.79.227.17:4971 , piece_id=NK55UPWQTKDWAAMPQTJPPOMZXH5VQGQ3QHJKRNMIRPFLFKQBZZIA, seg_pos={0 0}, upload=668]: Long tail cancellation due to optimal threshold Oct 27 15:28:49.842335229 (46m7.191642535s) [noise=true , address=145.79.227.10:4401 , piece_id=SZNSLFKN44YVSUHRXZTF6BBVPYQODYVEBWQF3JHP6ZDA2A723WAA, seg_pos={0 0}, upload=668]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1): protocol: write tcp 188.166.225.170:50900->145.79.227.10:4401: use of closed network connection; write tcp 188.166.225.170:50900->145.79.227.10:4401: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50900->145.79.227.10:4401: use of closed network connection Oct 27 15:28:49.842355578 (46m7.191662874s) [noise=true , address=145.79.227.10:4401 , piece_id=SZNSLFKN44YVSUHRXZTF6BBVPYQODYVEBWQF3JHP6ZDA2A723WAA, seg_pos={0 0}, upload=668]: Long tail cancellation due to optimal threshold Oct 27 15:28:49.842403068 (46m7.191710354s) [noise=true , address=145.79.227.19:4079 , piece_id=QXNYHJS2SBTG2QRH7AKW7NZ6SJG7MFO5DHXCMBZHGT6PJTOZ526Q, seg_pos={0 0}, upload=668]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud): protocol: write tcp 188.166.225.170:37220->145.79.227.19:4079: use of closed network connection; write tcp 188.166.225.170:37220->145.79.227.19:4079: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37220->145.79.227.19:4079: use of closed network connection Oct 27 15:28:49.842411143 (46m7.191718419s) [noise=true , address=145.79.227.19:4079 , piece_id=QXNYHJS2SBTG2QRH7AKW7NZ6SJG7MFO5DHXCMBZHGT6PJTOZ526Q, seg_pos={0 0}, upload=668]: Long tail cancellation due to optimal threshold Oct 27 15:28:49.842428026 (46m7.191735312s) [seg_pos={0 0}, upload=668]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 27 15:28:49.842507998 (46m7.191815284s) [upload=668]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 27 15:28:49.842513478 (46m7.191820764s) [upload=668]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 27 15:28:50.170126807 (46m7.519434093s) [upload=668]: Done uploading segments... Oct 27 15:28:50.170148138 (46m7.519455424s) [upload=668]: Upload finished. err: <nil> Duration: 5.968976263s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 3/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=3) [.....................X..............XX................X.....] duration: 3.390586402s FAIL (2025-10-27 15:33:21 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.390613754s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 5.983307671s FAIL (2025-10-27 15:33:06 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, Piece ID: 76MPB5O5XOR54TOXYCUNK2RBQMWYTWADCBAZV2KQYJ5K7XXDVFMQ) context deadline exceeded; piecestore: (Node ID: 12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, Piece ID: GR3XECVFMZ5E3HDMKK2HLKJREMIOZGWIUI7J3NSK7QGB523G7YJQ) context deadline exceeded; piecestore: (Node ID: 12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, Piece ID: FHAUWKUQAHGYR6ZJHKBCXT3ZPBG42MMJSADUGU7CIFUO3JPPNKKA) context deadline exceeded; piecestore: (Node ID: 1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV, Piece ID: 2EGSCSNCQJFXYT54PWEM33SP3RUYICBKKE7IDRQNMZ2ONPM6I5UA) context deadline exceeded ======LOGS====== ]: Done starting upload Oct 27 15:28:22.900340512 (45m40.249647808s) [upload=668]: Uploading segments... Oct 27 15:28:22.900564057 (45m40.249871343s) [upload=668]: Got next segment. Inline: false Oct 27 15:28:22.900613051 (45m40.249920337s) [upload=668]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 27 15:28:22.900620896 (45m40.249928182s) [upload=668]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 27 15:28:23.560720563 (45m40.91002784s) [seg_pos={0 0}, upload=668]: Begin upload segment... Oct 27 15:28:23.560802119 (45m40.910109405s) [seg_pos={0 0}, upload=668]: Done begin upload segment. Oct 27 15:28:23.560839660 (45m40.910146947s) [noise=true , address=145.79.227.20:4437 , piece_id=2EGSCSNCQJFXYT54PWEM33SP3RUYICBKKE7IDRQNMZ2ONPM6I5UA, seg_pos={0 0}, upload=668]: Uploading piece... Oct 27 15:28:23.560885057 (45m40.910192353s) [noise=true , address=145.79.227.19:4079 , piece_id=QXNYHJS2SBTG2QRH7AKW7NZ6SJG7MFO5DHXCMBZHGT6PJTOZ526Q, seg_pos={0 0}, upload=668]: Uploading piece... Oct 27 15:28:23.561607883 (45m40.910915169s) [noise=true , address=145.79.227.26:4653 , piece_id=MXREE4NCFOGYEHVQNFHZYD5DLD4PHBJS64JYSLTXXJZG7AC3LH6Q, seg_pos={0 0}, upload=668]: Uploading piece... Oct 27 15:28:23.561670783 (45m40.910978069s) [noise=true , address=77.90.31.10:4518 , piece_id=76MPB5O5XOR54TOXYCUNK2RBQMWYTWADCBAZV2KQYJ5K7XXDVFMQ, seg_pos={0 0}, upload=668]: Uploading piece... Oct 27 15:28:23.561966716 (45m40.911274002s) [noise=true , address=145.79.227.10:4401 , piece_id=SZNSLFKN44YVSUHRXZTF6BBVPYQODYVEBWQF3JHP6ZDA2A723WAA, seg_pos={0 0}, upload=668]: Uploading piece... Oct 27 15:28:23.562189701 (45m40.911496987s) [noise=true , address=145.79.227.4:4186 , piece_id=6APC73J3JHGVX3ZNCN3UL5DLHXRYORPQT6AYPFPX6ZQLQIRF5TNA, seg_pos={0 0}, upload=668]: Uploading piece... Oct 27 15:28:23.562277909 (45m40.911585195s) [noise=true , address=145.79.227.7:4294 , piece_id=QFLUZYLIGHO5E3ABEA5LFJPHAKDOF3J55Q4BRF5NXVKDNEHV5DBA, seg_pos={0 0}, upload=668]: Uploading piece... Oct 27 15:28:23.562572600 (45m40.911879887s) [noise=true , address=145.79.227.5:4222 , piece_id=B7W6PCNLLAPHXL7XOSJXEOVX6XZGNFDJWXR7MVYDGCONQCN55HYA, seg_pos={0 0}, upload=668]: Uploading piece... Oct 27 15:28:23.562586527 (45m40.911893813s) [noise=true , address=145.79.227.9:4366 , piece_id=5XLRE4OGUCQGZA5UXVNTNTFAMAGGQNHRR4MX23L3362K2DZLDREQ, seg_pos={0 0}, upload=668]: Uploading piece... Oct 27 15:28:23.562888732 (45m40.912196018s) [noise=true , address=145.79.227.21:4475 , piece_id=VDOFRCJKDQLXV4CBM4MA3UGK2PZMRJ3BW6H7PKJRZMZXW56Y5G4A, seg_pos={0 0}, upload=668]: Uploading piece... Oct 27 15:28:23.562953846 (45m40.912261123s) [noise=true , address=145.79.227.17:4971 , piece_id=NK55UPWQTKDWAAMPQTJPPOMZXH5VQGQ3QHJKRNMIRPFLFKQBZZIA, seg_pos={0 0}, upload=668]: Uploading piece... Oct 27 15:28:23.563182882 (45m40.912490158s) [noise=true , address=77.90.31.18:4807 , piece_id=OPDBUFUCTPZN2HH2SORHJL5L4EX2VWXATEFEIIEN5QTBQQFRGSZA, seg_pos={0 0}, upload=668]: Uploading piece... Oct 27 15:28:23.563335423 (45m40.912642709s) [noise=true , address=145.79.227.24:4582 , piece_id=K7COJQ2UG3CTGF4JVO27PTGWY4E3XGGRQWKC3LW24Y565B52YJCQ, seg_pos={0 0}, upload=668]: Uploading piece... Oct 27 15:28:23.563479036 (45m40.912786323s) [noise=true , address=77.90.31.13:4626 , piece_id=GR3XECVFMZ5E3HDMKK2HLKJREMIOZGWIUI7J3NSK7QGB523G7YJQ, seg_pos={0 0}, upload=668]: Uploading piece... Oct 27 15:28:23.563711468 (45m40.913018754s) [noise=true , address=45.91.116.5:4298 , piece_id=ATXRIPPIX6BAHUXCENJVDJ25K2Z6UH3QYL4MTBWPZKP7LFGHBX6A, seg_pos={0 0}, upload=668]: Uploading piece... Oct 27 15:28:23.564137260 (45m40.913444546s) [noise=true , address=145.79.227.5:4223 , piece_id=FHAUWKUQAHGYR6ZJHKBCXT3ZPBG42MMJSADUGU7CIFUO3JPPNKKA, seg_pos={0 0}, upload=668]: Uploading piece... Oct 27 15:28:27.241369967 (45m44.590677243s) [upload=668]: Next returned nil segment Oct 27 15:28:27.241385446 (45m44.590692732s) [upload=668]: Waiting for error group managing segments... Oct 27 15:28:29.744963349 (45m47.094270636s) [noise=true , address=145.79.227.5:4222 , piece_id=B7W6PCNLLAPHXL7XOSJXEOVX6XZGNFDJWXR7MVYDGCONQCN55HYA, seg_pos={0 0}, upload=668]: Done uploading piece. err: <nil> Oct 27 15:28:29.821202504 (45m47.17050978s) [noise=true , address=145.79.227.21:4475 , piece_id=VDOFRCJKDQLXV4CBM4MA3UGK2PZMRJ3BW6H7PKJRZMZXW56Y5G4A, seg_pos={0 0}, upload=668]: Done uploading piece. err: <nil> Oct 27 15:28:32.126653324 (45m49.47596061s) [noise=true , address=77.90.31.18:4807 , piece_id=OPDBUFUCTPZN2HH2SORHJL5L4EX2VWXATEFEIIEN5QTBQQFRGSZA, seg_pos={0 0}, upload=668]: Done uploading piece. err: <nil> Oct 27 15:28:33.087272849 (45m50.436580125s) [noise=true , address=45.91.116.5:4298 , piece_id=ATXRIPPIX6BAHUXCENJVDJ25K2Z6UH3QYL4MTBWPZKP7LFGHBX6A, seg_pos={0 0}, upload=668]: Done uploading piece. err: <nil> Oct 27 15:28:38.581899084 (45m55.93120637s) [noise=true , address=145.79.227.26:4653 , piece_id=MXREE4NCFOGYEHVQNFHZYD5DLD4PHBJS64JYSLTXXJZG7AC3LH6Q, seg_pos={0 0}, upload=668]: Done uploading piece. err: <nil> Oct 27 15:28:39.053322174 (45m56.40262946s) [noise=true , address=145.79.227.24:4582 , piece_id=K7COJQ2UG3CTGF4JVO27PTGWY4E3XGGRQWKC3LW24Y565B52YJCQ, seg_pos={0 0}, upload=668]: Done uploading piece. err: <nil> Oct 27 15:28:39.717124420 (45m57.066431696s) [noise=true , address=145.79.227.20:4437 , piece_id=2EGSCSNCQJFXYT54PWEM33SP3RUYICBKKE7IDRQNMZ2ONPM6I5UA, seg_pos={0 0}, upload=668]: Done uploading piece. err: <nil> Oct 27 15:28:41.618415745 (45m58.967723031s) [noise=true , address=77.90.31.13:4626 , piece_id=GR3XECVFMZ5E3HDMKK2HLKJREMIOZGWIUI7J3NSK7QGB523G7YJQ, seg_pos={0 0}, upload=668]: Done uploading piece. err: <nil> Oct 27 15:28:42.068020890 (45m59.417328166s) [noise=true , address=145.79.227.5:4223 , piece_id=FHAUWKUQAHGYR6ZJHKBCXT3ZPBG42MMJSADUGU7CIFUO3JPPNKKA, seg_pos={0 0}, upload=668]: Done uploading piece. err: <nil> Oct 27 15:28:42.150893235 (45m59.500200521s) [noise=true , address=77.90.31.10:4518 , piece_id=76MPB5O5XOR54TOXYCUNK2RBQMWYTWADCBAZV2KQYJ5K7XXDVFMQ, seg_pos={0 0}, upload=668]: Done uploading piece. err: <nil> Oct 27 15:28:43.703285619 (46m1.052592905s) [noise=true , address=145.79.227.9:4366 , piece_id=5XLRE4OGUCQGZA5UXVNTNTFAMAGGQNHRR4MX23L3362K2DZLDREQ, seg_pos={0 0}, upload=668]: Done uploading piece. err: <nil> Oct 27 15:28:49.699904003 (46m7.049211289s) [noise=true , address=145.79.227.4:4186 , piece_id=6APC73J3JHGVX3ZNCN3UL5DLHXRYORPQT6AYPFPX6ZQLQIRF5TNA, seg_pos={0 0}, upload=668]: Done uploading piece. err: <nil> Oct 27 15:28:49.842045738 (46m7.191353014s) [noise=true , address=145.79.227.7:4294 , piece_id=QFLUZYLIGHO5E3ABEA5LFJPHAKDOF3J55Q4BRF5NXVKDNEHV5DBA, seg_pos={0 0}, upload=668]: Done uploading piece. err: <nil> Oct 27 15:28:49.842081656 (46m7.191388932s) [seg_pos={0 0}, upload=668]: Segment reached optimal threshold of 13 pieces. Oct 27 15:28:49.842296495 (46m7.191603781s) [noise=true , address=145.79.227.17:4971 , piece_id=NK55UPWQTKDWAAMPQTJPPOMZXH5VQGQ3QHJKRNMIRPFLFKQBZZIA, seg_pos={0 0}, upload=668]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo): protocol: write tcp 188.166.225.170:55424->145.79.227.17:4971: use of closed network connection; write tcp 188.166.225.170:55424->145.79.227.17:4971: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55424->145.79.227.17:4971: use of closed network connection Oct 27 15:28:49.842306875 (46m7.191614161s) [noise=true , address=145.79.227.17:4971 , piece_id=NK55UPWQTKDWAAMPQTJPPOMZXH5VQGQ3QHJKRNMIRPFLFKQBZZIA, seg_pos={0 0}, upload=668]: Long tail cancellation due to optimal threshold Oct 27 15:28:49.842335229 (46m7.191642535s) [noise=true , address=145.79.227.10:4401 , piece_id=SZNSLFKN44YVSUHRXZTF6BBVPYQODYVEBWQF3JHP6ZDA2A723WAA, seg_pos={0 0}, upload=668]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1): protocol: write tcp 188.166.225.170:50900->145.79.227.10:4401: use of closed network connection; write tcp 188.166.225.170:50900->145.79.227.10:4401: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50900->145.79.227.10:4401: use of closed network connection Oct 27 15:28:49.842355578 (46m7.191662874s) [noise=true , address=145.79.227.10:4401 , piece_id=SZNSLFKN44YVSUHRXZTF6BBVPYQODYVEBWQF3JHP6ZDA2A723WAA, seg_pos={0 0}, upload=668]: Long tail cancellation due to optimal threshold Oct 27 15:28:49.842403068 (46m7.191710354s) [noise=true , address=145.79.227.19:4079 , piece_id=QXNYHJS2SBTG2QRH7AKW7NZ6SJG7MFO5DHXCMBZHGT6PJTOZ526Q, seg_pos={0 0}, upload=668]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud): protocol: write tcp 188.166.225.170:37220->145.79.227.19:4079: use of closed network connection; write tcp 188.166.225.170:37220->145.79.227.19:4079: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37220->145.79.227.19:4079: use of closed network connection Oct 27 15:28:49.842411143 (46m7.191718419s) [noise=true , address=145.79.227.19:4079 , piece_id=QXNYHJS2SBTG2QRH7AKW7NZ6SJG7MFO5DHXCMBZHGT6PJTOZ526Q, seg_pos={0 0}, upload=668]: Long tail cancellation due to optimal threshold Oct 27 15:28:49.842428026 (46m7.191735312s) [seg_pos={0 0}, upload=668]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 27 15:28:49.842507998 (46m7.191815284s) [upload=668]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 27 15:28:49.842513478 (46m7.191820764s) [upload=668]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 27 15:28:50.170126807 (46m7.519434093s) [upload=668]: Done uploading segments... Oct 27 15:28:50.170148138 (46m7.519455424s) [upload=668]: Upload finished. err: <nil> Duration: 5.983334412s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 6.120680777s FAIL (2025-10-27 15:32:58 UTC) orbiter warning: context deadline exceeded Duration: 6.120704641s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 3/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=3) [......................X..............XX................X....] duration: 3.460739398s FAIL (2025-10-27 15:32:28 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.460764396s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 5.821349985s FAIL (2025-10-27 15:32:19 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, Piece ID: 76MPB5O5XOR54TOXYCUNK2RBQMWYTWADCBAZV2KQYJ5K7XXDVFMQ) context deadline exceeded; piecestore: (Node ID: 12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, Piece ID: GR3XECVFMZ5E3HDMKK2HLKJREMIOZGWIUI7J3NSK7QGB523G7YJQ) context deadline exceeded; piecestore: (Node ID: 12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, Piece ID: FHAUWKUQAHGYR6ZJHKBCXT3ZPBG42MMJSADUGU7CIFUO3JPPNKKA) context deadline exceeded; piecestore: (Node ID: 1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV, Piece ID: 2EGSCSNCQJFXYT54PWEM33SP3RUYICBKKE7IDRQNMZ2ONPM6I5UA) context deadline exceeded ======LOGS====== ]: Done starting upload Oct 27 15:28:22.900340512 (45m40.249647808s) [upload=668]: Uploading segments... Oct 27 15:28:22.900564057 (45m40.249871343s) [upload=668]: Got next segment. Inline: false Oct 27 15:28:22.900613051 (45m40.249920337s) [upload=668]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 27 15:28:22.900620896 (45m40.249928182s) [upload=668]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 27 15:28:23.560720563 (45m40.91002784s) [seg_pos={0 0}, upload=668]: Begin upload segment... Oct 27 15:28:23.560802119 (45m40.910109405s) [seg_pos={0 0}, upload=668]: Done begin upload segment. Oct 27 15:28:23.560839660 (45m40.910146947s) [noise=true , address=145.79.227.20:4437 , piece_id=2EGSCSNCQJFXYT54PWEM33SP3RUYICBKKE7IDRQNMZ2ONPM6I5UA, seg_pos={0 0}, upload=668]: Uploading piece... Oct 27 15:28:23.560885057 (45m40.910192353s) [noise=true , address=145.79.227.19:4079 , piece_id=QXNYHJS2SBTG2QRH7AKW7NZ6SJG7MFO5DHXCMBZHGT6PJTOZ526Q, seg_pos={0 0}, upload=668]: Uploading piece... Oct 27 15:28:23.561607883 (45m40.910915169s) [noise=true , address=145.79.227.26:4653 , piece_id=MXREE4NCFOGYEHVQNFHZYD5DLD4PHBJS64JYSLTXXJZG7AC3LH6Q, seg_pos={0 0}, upload=668]: Uploading piece... Oct 27 15:28:23.561670783 (45m40.910978069s) [noise=true , address=77.90.31.10:4518 , piece_id=76MPB5O5XOR54TOXYCUNK2RBQMWYTWADCBAZV2KQYJ5K7XXDVFMQ, seg_pos={0 0}, upload=668]: Uploading piece... Oct 27 15:28:23.561966716 (45m40.911274002s) [noise=true , address=145.79.227.10:4401 , piece_id=SZNSLFKN44YVSUHRXZTF6BBVPYQODYVEBWQF3JHP6ZDA2A723WAA, seg_pos={0 0}, upload=668]: Uploading piece... Oct 27 15:28:23.562189701 (45m40.911496987s) [noise=true , address=145.79.227.4:4186 , piece_id=6APC73J3JHGVX3ZNCN3UL5DLHXRYORPQT6AYPFPX6ZQLQIRF5TNA, seg_pos={0 0}, upload=668]: Uploading piece... Oct 27 15:28:23.562277909 (45m40.911585195s) [noise=true , address=145.79.227.7:4294 , piece_id=QFLUZYLIGHO5E3ABEA5LFJPHAKDOF3J55Q4BRF5NXVKDNEHV5DBA, seg_pos={0 0}, upload=668]: Uploading piece... Oct 27 15:28:23.562572600 (45m40.911879887s) [noise=true , address=145.79.227.5:4222 , piece_id=B7W6PCNLLAPHXL7XOSJXEOVX6XZGNFDJWXR7MVYDGCONQCN55HYA, seg_pos={0 0}, upload=668]: Uploading piece... Oct 27 15:28:23.562586527 (45m40.911893813s) [noise=true , address=145.79.227.9:4366 , piece_id=5XLRE4OGUCQGZA5UXVNTNTFAMAGGQNHRR4MX23L3362K2DZLDREQ, seg_pos={0 0}, upload=668]: Uploading piece... Oct 27 15:28:23.562888732 (45m40.912196018s) [noise=true , address=145.79.227.21:4475 , piece_id=VDOFRCJKDQLXV4CBM4MA3UGK2PZMRJ3BW6H7PKJRZMZXW56Y5G4A, seg_pos={0 0}, upload=668]: Uploading piece... Oct 27 15:28:23.562953846 (45m40.912261123s) [noise=true , address=145.79.227.17:4971 , piece_id=NK55UPWQTKDWAAMPQTJPPOMZXH5VQGQ3QHJKRNMIRPFLFKQBZZIA, seg_pos={0 0}, upload=668]: Uploading piece... Oct 27 15:28:23.563182882 (45m40.912490158s) [noise=true , address=77.90.31.18:4807 , piece_id=OPDBUFUCTPZN2HH2SORHJL5L4EX2VWXATEFEIIEN5QTBQQFRGSZA, seg_pos={0 0}, upload=668]: Uploading piece... Oct 27 15:28:23.563335423 (45m40.912642709s) [noise=true , address=145.79.227.24:4582 , piece_id=K7COJQ2UG3CTGF4JVO27PTGWY4E3XGGRQWKC3LW24Y565B52YJCQ, seg_pos={0 0}, upload=668]: Uploading piece... Oct 27 15:28:23.563479036 (45m40.912786323s) [noise=true , address=77.90.31.13:4626 , piece_id=GR3XECVFMZ5E3HDMKK2HLKJREMIOZGWIUI7J3NSK7QGB523G7YJQ, seg_pos={0 0}, upload=668]: Uploading piece... Oct 27 15:28:23.563711468 (45m40.913018754s) [noise=true , address=45.91.116.5:4298 , piece_id=ATXRIPPIX6BAHUXCENJVDJ25K2Z6UH3QYL4MTBWPZKP7LFGHBX6A, seg_pos={0 0}, upload=668]: Uploading piece... Oct 27 15:28:23.564137260 (45m40.913444546s) [noise=true , address=145.79.227.5:4223 , piece_id=FHAUWKUQAHGYR6ZJHKBCXT3ZPBG42MMJSADUGU7CIFUO3JPPNKKA, seg_pos={0 0}, upload=668]: Uploading piece... Oct 27 15:28:27.241369967 (45m44.590677243s) [upload=668]: Next returned nil segment Oct 27 15:28:27.241385446 (45m44.590692732s) [upload=668]: Waiting for error group managing segments... Oct 27 15:28:29.744963349 (45m47.094270636s) [noise=true , address=145.79.227.5:4222 , piece_id=B7W6PCNLLAPHXL7XOSJXEOVX6XZGNFDJWXR7MVYDGCONQCN55HYA, seg_pos={0 0}, upload=668]: Done uploading piece. err: <nil> Oct 27 15:28:29.821202504 (45m47.17050978s) [noise=true , address=145.79.227.21:4475 , piece_id=VDOFRCJKDQLXV4CBM4MA3UGK2PZMRJ3BW6H7PKJRZMZXW56Y5G4A, seg_pos={0 0}, upload=668]: Done uploading piece. err: <nil> Oct 27 15:28:32.126653324 (45m49.47596061s) [noise=true , address=77.90.31.18:4807 , piece_id=OPDBUFUCTPZN2HH2SORHJL5L4EX2VWXATEFEIIEN5QTBQQFRGSZA, seg_pos={0 0}, upload=668]: Done uploading piece. err: <nil> Oct 27 15:28:33.087272849 (45m50.436580125s) [noise=true , address=45.91.116.5:4298 , piece_id=ATXRIPPIX6BAHUXCENJVDJ25K2Z6UH3QYL4MTBWPZKP7LFGHBX6A, seg_pos={0 0}, upload=668]: Done uploading piece. err: <nil> Oct 27 15:28:38.581899084 (45m55.93120637s) [noise=true , address=145.79.227.26:4653 , piece_id=MXREE4NCFOGYEHVQNFHZYD5DLD4PHBJS64JYSLTXXJZG7AC3LH6Q, seg_pos={0 0}, upload=668]: Done uploading piece. err: <nil> Oct 27 15:28:39.053322174 (45m56.40262946s) [noise=true , address=145.79.227.24:4582 , piece_id=K7COJQ2UG3CTGF4JVO27PTGWY4E3XGGRQWKC3LW24Y565B52YJCQ, seg_pos={0 0}, upload=668]: Done uploading piece. err: <nil> Oct 27 15:28:39.717124420 (45m57.066431696s) [noise=true , address=145.79.227.20:4437 , piece_id=2EGSCSNCQJFXYT54PWEM33SP3RUYICBKKE7IDRQNMZ2ONPM6I5UA, seg_pos={0 0}, upload=668]: Done uploading piece. err: <nil> Oct 27 15:28:41.618415745 (45m58.967723031s) [noise=true , address=77.90.31.13:4626 , piece_id=GR3XECVFMZ5E3HDMKK2HLKJREMIOZGWIUI7J3NSK7QGB523G7YJQ, seg_pos={0 0}, upload=668]: Done uploading piece. err: <nil> Oct 27 15:28:42.068020890 (45m59.417328166s) [noise=true , address=145.79.227.5:4223 , piece_id=FHAUWKUQAHGYR6ZJHKBCXT3ZPBG42MMJSADUGU7CIFUO3JPPNKKA, seg_pos={0 0}, upload=668]: Done uploading piece. err: <nil> Oct 27 15:28:42.150893235 (45m59.500200521s) [noise=true , address=77.90.31.10:4518 , piece_id=76MPB5O5XOR54TOXYCUNK2RBQMWYTWADCBAZV2KQYJ5K7XXDVFMQ, seg_pos={0 0}, upload=668]: Done uploading piece. err: <nil> Oct 27 15:28:43.703285619 (46m1.052592905s) [noise=true , address=145.79.227.9:4366 , piece_id=5XLRE4OGUCQGZA5UXVNTNTFAMAGGQNHRR4MX23L3362K2DZLDREQ, seg_pos={0 0}, upload=668]: Done uploading piece. err: <nil> Oct 27 15:28:49.699904003 (46m7.049211289s) [noise=true , address=145.79.227.4:4186 , piece_id=6APC73J3JHGVX3ZNCN3UL5DLHXRYORPQT6AYPFPX6ZQLQIRF5TNA, seg_pos={0 0}, upload=668]: Done uploading piece. err: <nil> Oct 27 15:28:49.842045738 (46m7.191353014s) [noise=true , address=145.79.227.7:4294 , piece_id=QFLUZYLIGHO5E3ABEA5LFJPHAKDOF3J55Q4BRF5NXVKDNEHV5DBA, seg_pos={0 0}, upload=668]: Done uploading piece. err: <nil> Oct 27 15:28:49.842081656 (46m7.191388932s) [seg_pos={0 0}, upload=668]: Segment reached optimal threshold of 13 pieces. Oct 27 15:28:49.842296495 (46m7.191603781s) [noise=true , address=145.79.227.17:4971 , piece_id=NK55UPWQTKDWAAMPQTJPPOMZXH5VQGQ3QHJKRNMIRPFLFKQBZZIA, seg_pos={0 0}, upload=668]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo): protocol: write tcp 188.166.225.170:55424->145.79.227.17:4971: use of closed network connection; write tcp 188.166.225.170:55424->145.79.227.17:4971: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55424->145.79.227.17:4971: use of closed network connection Oct 27 15:28:49.842306875 (46m7.191614161s) [noise=true , address=145.79.227.17:4971 , piece_id=NK55UPWQTKDWAAMPQTJPPOMZXH5VQGQ3QHJKRNMIRPFLFKQBZZIA, seg_pos={0 0}, upload=668]: Long tail cancellation due to optimal threshold Oct 27 15:28:49.842335229 (46m7.191642535s) [noise=true , address=145.79.227.10:4401 , piece_id=SZNSLFKN44YVSUHRXZTF6BBVPYQODYVEBWQF3JHP6ZDA2A723WAA, seg_pos={0 0}, upload=668]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1): protocol: write tcp 188.166.225.170:50900->145.79.227.10:4401: use of closed network connection; write tcp 188.166.225.170:50900->145.79.227.10:4401: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50900->145.79.227.10:4401: use of closed network connection Oct 27 15:28:49.842355578 (46m7.191662874s) [noise=true , address=145.79.227.10:4401 , piece_id=SZNSLFKN44YVSUHRXZTF6BBVPYQODYVEBWQF3JHP6ZDA2A723WAA, seg_pos={0 0}, upload=668]: Long tail cancellation due to optimal threshold Oct 27 15:28:49.842403068 (46m7.191710354s) [noise=true , address=145.79.227.19:4079 , piece_id=QXNYHJS2SBTG2QRH7AKW7NZ6SJG7MFO5DHXCMBZHGT6PJTOZ526Q, seg_pos={0 0}, upload=668]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud): protocol: write tcp 188.166.225.170:37220->145.79.227.19:4079: use of closed network connection; write tcp 188.166.225.170:37220->145.79.227.19:4079: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37220->145.79.227.19:4079: use of closed network connection Oct 27 15:28:49.842411143 (46m7.191718419s) [noise=true , address=145.79.227.19:4079 , piece_id=QXNYHJS2SBTG2QRH7AKW7NZ6SJG7MFO5DHXCMBZHGT6PJTOZ526Q, seg_pos={0 0}, upload=668]: Long tail cancellation due to optimal threshold Oct 27 15:28:49.842428026 (46m7.191735312s) [seg_pos={0 0}, upload=668]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 27 15:28:49.842507998 (46m7.191815284s) [upload=668]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 27 15:28:49.842513478 (46m7.191820764s) [upload=668]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 27 15:28:50.170126807 (46m7.519434093s) [upload=668]: Done uploading segments... Oct 27 15:28:50.170148138 (46m7.519455424s) [upload=668]: Upload finished. err: <nil> Duration: 5.821383819s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 6.160926844s FAIL (2025-10-27 15:31:46 UTC) orbiter warning: context deadline exceeded Duration: 6.160947494s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 3/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=3) [.......................X..............XX................X...] duration: 3.464759174s FAIL (2025-10-27 15:31:40 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.464796325s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 6.111521146s FAIL (2025-10-27 15:31:35 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, Piece ID: 76MPB5O5XOR54TOXYCUNK2RBQMWYTWADCBAZV2KQYJ5K7XXDVFMQ) context deadline exceeded; piecestore: (Node ID: 12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, Piece ID: GR3XECVFMZ5E3HDMKK2HLKJREMIOZGWIUI7J3NSK7QGB523G7YJQ) context deadline exceeded; piecestore: (Node ID: 12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, Piece ID: FHAUWKUQAHGYR6ZJHKBCXT3ZPBG42MMJSADUGU7CIFUO3JPPNKKA) context deadline exceeded; piecestore: (Node ID: 1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV, Piece ID: 2EGSCSNCQJFXYT54PWEM33SP3RUYICBKKE7IDRQNMZ2ONPM6I5UA) context deadline exceeded ======LOGS====== ]: Done starting upload Oct 27 15:28:22.900340512 (45m40.249647808s) [upload=668]: Uploading segments... Oct 27 15:28:22.900564057 (45m40.249871343s) [upload=668]: Got next segment. Inline: false Oct 27 15:28:22.900613051 (45m40.249920337s) [upload=668]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 27 15:28:22.900620896 (45m40.249928182s) [upload=668]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 27 15:28:23.560720563 (45m40.91002784s) [seg_pos={0 0}, upload=668]: Begin upload segment... Oct 27 15:28:23.560802119 (45m40.910109405s) [seg_pos={0 0}, upload=668]: Done begin upload segment. Oct 27 15:28:23.560839660 (45m40.910146947s) [noise=true , address=145.79.227.20:4437 , piece_id=2EGSCSNCQJFXYT54PWEM33SP3RUYICBKKE7IDRQNMZ2ONPM6I5UA, seg_pos={0 0}, upload=668]: Uploading piece... Oct 27 15:28:23.560885057 (45m40.910192353s) [noise=true , address=145.79.227.19:4079 , piece_id=QXNYHJS2SBTG2QRH7AKW7NZ6SJG7MFO5DHXCMBZHGT6PJTOZ526Q, seg_pos={0 0}, upload=668]: Uploading piece... Oct 27 15:28:23.561607883 (45m40.910915169s) [noise=true , address=145.79.227.26:4653 , piece_id=MXREE4NCFOGYEHVQNFHZYD5DLD4PHBJS64JYSLTXXJZG7AC3LH6Q, seg_pos={0 0}, upload=668]: Uploading piece... Oct 27 15:28:23.561670783 (45m40.910978069s) [noise=true , address=77.90.31.10:4518 , piece_id=76MPB5O5XOR54TOXYCUNK2RBQMWYTWADCBAZV2KQYJ5K7XXDVFMQ, seg_pos={0 0}, upload=668]: Uploading piece... Oct 27 15:28:23.561966716 (45m40.911274002s) [noise=true , address=145.79.227.10:4401 , piece_id=SZNSLFKN44YVSUHRXZTF6BBVPYQODYVEBWQF3JHP6ZDA2A723WAA, seg_pos={0 0}, upload=668]: Uploading piece... Oct 27 15:28:23.562189701 (45m40.911496987s) [noise=true , address=145.79.227.4:4186 , piece_id=6APC73J3JHGVX3ZNCN3UL5DLHXRYORPQT6AYPFPX6ZQLQIRF5TNA, seg_pos={0 0}, upload=668]: Uploading piece... Oct 27 15:28:23.562277909 (45m40.911585195s) [noise=true , address=145.79.227.7:4294 , piece_id=QFLUZYLIGHO5E3ABEA5LFJPHAKDOF3J55Q4BRF5NXVKDNEHV5DBA, seg_pos={0 0}, upload=668]: Uploading piece... Oct 27 15:28:23.562572600 (45m40.911879887s) [noise=true , address=145.79.227.5:4222 , piece_id=B7W6PCNLLAPHXL7XOSJXEOVX6XZGNFDJWXR7MVYDGCONQCN55HYA, seg_pos={0 0}, upload=668]: Uploading piece... Oct 27 15:28:23.562586527 (45m40.911893813s) [noise=true , address=145.79.227.9:4366 , piece_id=5XLRE4OGUCQGZA5UXVNTNTFAMAGGQNHRR4MX23L3362K2DZLDREQ, seg_pos={0 0}, upload=668]: Uploading piece... Oct 27 15:28:23.562888732 (45m40.912196018s) [noise=true , address=145.79.227.21:4475 , piece_id=VDOFRCJKDQLXV4CBM4MA3UGK2PZMRJ3BW6H7PKJRZMZXW56Y5G4A, seg_pos={0 0}, upload=668]: Uploading piece... Oct 27 15:28:23.562953846 (45m40.912261123s) [noise=true , address=145.79.227.17:4971 , piece_id=NK55UPWQTKDWAAMPQTJPPOMZXH5VQGQ3QHJKRNMIRPFLFKQBZZIA, seg_pos={0 0}, upload=668]: Uploading piece... Oct 27 15:28:23.563182882 (45m40.912490158s) [noise=true , address=77.90.31.18:4807 , piece_id=OPDBUFUCTPZN2HH2SORHJL5L4EX2VWXATEFEIIEN5QTBQQFRGSZA, seg_pos={0 0}, upload=668]: Uploading piece... Oct 27 15:28:23.563335423 (45m40.912642709s) [noise=true , address=145.79.227.24:4582 , piece_id=K7COJQ2UG3CTGF4JVO27PTGWY4E3XGGRQWKC3LW24Y565B52YJCQ, seg_pos={0 0}, upload=668]: Uploading piece... Oct 27 15:28:23.563479036 (45m40.912786323s) [noise=true , address=77.90.31.13:4626 , piece_id=GR3XECVFMZ5E3HDMKK2HLKJREMIOZGWIUI7J3NSK7QGB523G7YJQ, seg_pos={0 0}, upload=668]: Uploading piece... Oct 27 15:28:23.563711468 (45m40.913018754s) [noise=true , address=45.91.116.5:4298 , piece_id=ATXRIPPIX6BAHUXCENJVDJ25K2Z6UH3QYL4MTBWPZKP7LFGHBX6A, seg_pos={0 0}, upload=668]: Uploading piece... Oct 27 15:28:23.564137260 (45m40.913444546s) [noise=true , address=145.79.227.5:4223 , piece_id=FHAUWKUQAHGYR6ZJHKBCXT3ZPBG42MMJSADUGU7CIFUO3JPPNKKA, seg_pos={0 0}, upload=668]: Uploading piece... Oct 27 15:28:27.241369967 (45m44.590677243s) [upload=668]: Next returned nil segment Oct 27 15:28:27.241385446 (45m44.590692732s) [upload=668]: Waiting for error group managing segments... Oct 27 15:28:29.744963349 (45m47.094270636s) [noise=true , address=145.79.227.5:4222 , piece_id=B7W6PCNLLAPHXL7XOSJXEOVX6XZGNFDJWXR7MVYDGCONQCN55HYA, seg_pos={0 0}, upload=668]: Done uploading piece. err: <nil> Oct 27 15:28:29.821202504 (45m47.17050978s) [noise=true , address=145.79.227.21:4475 , piece_id=VDOFRCJKDQLXV4CBM4MA3UGK2PZMRJ3BW6H7PKJRZMZXW56Y5G4A, seg_pos={0 0}, upload=668]: Done uploading piece. err: <nil> Oct 27 15:28:32.126653324 (45m49.47596061s) [noise=true , address=77.90.31.18:4807 , piece_id=OPDBUFUCTPZN2HH2SORHJL5L4EX2VWXATEFEIIEN5QTBQQFRGSZA, seg_pos={0 0}, upload=668]: Done uploading piece. err: <nil> Oct 27 15:28:33.087272849 (45m50.436580125s) [noise=true , address=45.91.116.5:4298 , piece_id=ATXRIPPIX6BAHUXCENJVDJ25K2Z6UH3QYL4MTBWPZKP7LFGHBX6A, seg_pos={0 0}, upload=668]: Done uploading piece. err: <nil> Oct 27 15:28:38.581899084 (45m55.93120637s) [noise=true , address=145.79.227.26:4653 , piece_id=MXREE4NCFOGYEHVQNFHZYD5DLD4PHBJS64JYSLTXXJZG7AC3LH6Q, seg_pos={0 0}, upload=668]: Done uploading piece. err: <nil> Oct 27 15:28:39.053322174 (45m56.40262946s) [noise=true , address=145.79.227.24:4582 , piece_id=K7COJQ2UG3CTGF4JVO27PTGWY4E3XGGRQWKC3LW24Y565B52YJCQ, seg_pos={0 0}, upload=668]: Done uploading piece. err: <nil> Oct 27 15:28:39.717124420 (45m57.066431696s) [noise=true , address=145.79.227.20:4437 , piece_id=2EGSCSNCQJFXYT54PWEM33SP3RUYICBKKE7IDRQNMZ2ONPM6I5UA, seg_pos={0 0}, upload=668]: Done uploading piece. err: <nil> Oct 27 15:28:41.618415745 (45m58.967723031s) [noise=true , address=77.90.31.13:4626 , piece_id=GR3XECVFMZ5E3HDMKK2HLKJREMIOZGWIUI7J3NSK7QGB523G7YJQ, seg_pos={0 0}, upload=668]: Done uploading piece. err: <nil> Oct 27 15:28:42.068020890 (45m59.417328166s) [noise=true , address=145.79.227.5:4223 , piece_id=FHAUWKUQAHGYR6ZJHKBCXT3ZPBG42MMJSADUGU7CIFUO3JPPNKKA, seg_pos={0 0}, upload=668]: Done uploading piece. err: <nil> Oct 27 15:28:42.150893235 (45m59.500200521s) [noise=true , address=77.90.31.10:4518 , piece_id=76MPB5O5XOR54TOXYCUNK2RBQMWYTWADCBAZV2KQYJ5K7XXDVFMQ, seg_pos={0 0}, upload=668]: Done uploading piece. err: <nil> Oct 27 15:28:43.703285619 (46m1.052592905s) [noise=true , address=145.79.227.9:4366 , piece_id=5XLRE4OGUCQGZA5UXVNTNTFAMAGGQNHRR4MX23L3362K2DZLDREQ, seg_pos={0 0}, upload=668]: Done uploading piece. err: <nil> Oct 27 15:28:49.699904003 (46m7.049211289s) [noise=true , address=145.79.227.4:4186 , piece_id=6APC73J3JHGVX3ZNCN3UL5DLHXRYORPQT6AYPFPX6ZQLQIRF5TNA, seg_pos={0 0}, upload=668]: Done uploading piece. err: <nil> Oct 27 15:28:49.842045738 (46m7.191353014s) [noise=true , address=145.79.227.7:4294 , piece_id=QFLUZYLIGHO5E3ABEA5LFJPHAKDOF3J55Q4BRF5NXVKDNEHV5DBA, seg_pos={0 0}, upload=668]: Done uploading piece. err: <nil> Oct 27 15:28:49.842081656 (46m7.191388932s) [seg_pos={0 0}, upload=668]: Segment reached optimal threshold of 13 pieces. Oct 27 15:28:49.842296495 (46m7.191603781s) [noise=true , address=145.79.227.17:4971 , piece_id=NK55UPWQTKDWAAMPQTJPPOMZXH5VQGQ3QHJKRNMIRPFLFKQBZZIA, seg_pos={0 0}, upload=668]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo): protocol: write tcp 188.166.225.170:55424->145.79.227.17:4971: use of closed network connection; write tcp 188.166.225.170:55424->145.79.227.17:4971: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55424->145.79.227.17:4971: use of closed network connection Oct 27 15:28:49.842306875 (46m7.191614161s) [noise=true , address=145.79.227.17:4971 , piece_id=NK55UPWQTKDWAAMPQTJPPOMZXH5VQGQ3QHJKRNMIRPFLFKQBZZIA, seg_pos={0 0}, upload=668]: Long tail cancellation due to optimal threshold Oct 27 15:28:49.842335229 (46m7.191642535s) [noise=true , address=145.79.227.10:4401 , piece_id=SZNSLFKN44YVSUHRXZTF6BBVPYQODYVEBWQF3JHP6ZDA2A723WAA, seg_pos={0 0}, upload=668]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1): protocol: write tcp 188.166.225.170:50900->145.79.227.10:4401: use of closed network connection; write tcp 188.166.225.170:50900->145.79.227.10:4401: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50900->145.79.227.10:4401: use of closed network connection Oct 27 15:28:49.842355578 (46m7.191662874s) [noise=true , address=145.79.227.10:4401 , piece_id=SZNSLFKN44YVSUHRXZTF6BBVPYQODYVEBWQF3JHP6ZDA2A723WAA, seg_pos={0 0}, upload=668]: Long tail cancellation due to optimal threshold Oct 27 15:28:49.842403068 (46m7.191710354s) [noise=true , address=145.79.227.19:4079 , piece_id=QXNYHJS2SBTG2QRH7AKW7NZ6SJG7MFO5DHXCMBZHGT6PJTOZ526Q, seg_pos={0 0}, upload=668]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud): protocol: write tcp 188.166.225.170:37220->145.79.227.19:4079: use of closed network connection; write tcp 188.166.225.170:37220->145.79.227.19:4079: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37220->145.79.227.19:4079: use of closed network connection Oct 27 15:28:49.842411143 (46m7.191718419s) [noise=true , address=145.79.227.19:4079 , piece_id=QXNYHJS2SBTG2QRH7AKW7NZ6SJG7MFO5DHXCMBZHGT6PJTOZ526Q, seg_pos={0 0}, upload=668]: Long tail cancellation due to optimal threshold Oct 27 15:28:49.842428026 (46m7.191735312s) [seg_pos={0 0}, upload=668]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 27 15:28:49.842507998 (46m7.191815284s) [upload=668]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 27 15:28:49.842513478 (46m7.191820764s) [upload=668]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 27 15:28:50.170126807 (46m7.519434093s) [upload=668]: Done uploading segments... Oct 27 15:28:50.170148138 (46m7.519455424s) [upload=668]: Upload finished. err: <nil> Duration: 6.111547545s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 5.952610879s FAIL (2025-10-27 15:30:34 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, Piece ID: 76MPB5O5XOR54TOXYCUNK2RBQMWYTWADCBAZV2KQYJ5K7XXDVFMQ) context deadline exceeded; piecestore: (Node ID: 12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, Piece ID: GR3XECVFMZ5E3HDMKK2HLKJREMIOZGWIUI7J3NSK7QGB523G7YJQ) context deadline exceeded; piecestore: (Node ID: 12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, Piece ID: FHAUWKUQAHGYR6ZJHKBCXT3ZPBG42MMJSADUGU7CIFUO3JPPNKKA) context deadline exceeded; piecestore: (Node ID: 1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV, Piece ID: 2EGSCSNCQJFXYT54PWEM33SP3RUYICBKKE7IDRQNMZ2ONPM6I5UA) context deadline exceeded ======LOGS====== ]: Done starting upload Oct 27 15:28:22.900340512 (45m40.249647808s) [upload=668]: Uploading segments... Oct 27 15:28:22.900564057 (45m40.249871343s) [upload=668]: Got next segment. Inline: false Oct 27 15:28:22.900613051 (45m40.249920337s) [upload=668]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 27 15:28:22.900620896 (45m40.249928182s) [upload=668]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 27 15:28:23.560720563 (45m40.91002784s) [seg_pos={0 0}, upload=668]: Begin upload segment... Oct 27 15:28:23.560802119 (45m40.910109405s) [seg_pos={0 0}, upload=668]: Done begin upload segment. Oct 27 15:28:23.560839660 (45m40.910146947s) [noise=true , address=145.79.227.20:4437 , piece_id=2EGSCSNCQJFXYT54PWEM33SP3RUYICBKKE7IDRQNMZ2ONPM6I5UA, seg_pos={0 0}, upload=668]: Uploading piece... Oct 27 15:28:23.560885057 (45m40.910192353s) [noise=true , address=145.79.227.19:4079 , piece_id=QXNYHJS2SBTG2QRH7AKW7NZ6SJG7MFO5DHXCMBZHGT6PJTOZ526Q, seg_pos={0 0}, upload=668]: Uploading piece... Oct 27 15:28:23.561607883 (45m40.910915169s) [noise=true , address=145.79.227.26:4653 , piece_id=MXREE4NCFOGYEHVQNFHZYD5DLD4PHBJS64JYSLTXXJZG7AC3LH6Q, seg_pos={0 0}, upload=668]: Uploading piece... Oct 27 15:28:23.561670783 (45m40.910978069s) [noise=true , address=77.90.31.10:4518 , piece_id=76MPB5O5XOR54TOXYCUNK2RBQMWYTWADCBAZV2KQYJ5K7XXDVFMQ, seg_pos={0 0}, upload=668]: Uploading piece... Oct 27 15:28:23.561966716 (45m40.911274002s) [noise=true , address=145.79.227.10:4401 , piece_id=SZNSLFKN44YVSUHRXZTF6BBVPYQODYVEBWQF3JHP6ZDA2A723WAA, seg_pos={0 0}, upload=668]: Uploading piece... Oct 27 15:28:23.562189701 (45m40.911496987s) [noise=true , address=145.79.227.4:4186 , piece_id=6APC73J3JHGVX3ZNCN3UL5DLHXRYORPQT6AYPFPX6ZQLQIRF5TNA, seg_pos={0 0}, upload=668]: Uploading piece... Oct 27 15:28:23.562277909 (45m40.911585195s) [noise=true , address=145.79.227.7:4294 , piece_id=QFLUZYLIGHO5E3ABEA5LFJPHAKDOF3J55Q4BRF5NXVKDNEHV5DBA, seg_pos={0 0}, upload=668]: Uploading piece... Oct 27 15:28:23.562572600 (45m40.911879887s) [noise=true , address=145.79.227.5:4222 , piece_id=B7W6PCNLLAPHXL7XOSJXEOVX6XZGNFDJWXR7MVYDGCONQCN55HYA, seg_pos={0 0}, upload=668]: Uploading piece... Oct 27 15:28:23.562586527 (45m40.911893813s) [noise=true , address=145.79.227.9:4366 , piece_id=5XLRE4OGUCQGZA5UXVNTNTFAMAGGQNHRR4MX23L3362K2DZLDREQ, seg_pos={0 0}, upload=668]: Uploading piece... Oct 27 15:28:23.562888732 (45m40.912196018s) [noise=true , address=145.79.227.21:4475 , piece_id=VDOFRCJKDQLXV4CBM4MA3UGK2PZMRJ3BW6H7PKJRZMZXW56Y5G4A, seg_pos={0 0}, upload=668]: Uploading piece... Oct 27 15:28:23.562953846 (45m40.912261123s) [noise=true , address=145.79.227.17:4971 , piece_id=NK55UPWQTKDWAAMPQTJPPOMZXH5VQGQ3QHJKRNMIRPFLFKQBZZIA, seg_pos={0 0}, upload=668]: Uploading piece... Oct 27 15:28:23.563182882 (45m40.912490158s) [noise=true , address=77.90.31.18:4807 , piece_id=OPDBUFUCTPZN2HH2SORHJL5L4EX2VWXATEFEIIEN5QTBQQFRGSZA, seg_pos={0 0}, upload=668]: Uploading piece... Oct 27 15:28:23.563335423 (45m40.912642709s) [noise=true , address=145.79.227.24:4582 , piece_id=K7COJQ2UG3CTGF4JVO27PTGWY4E3XGGRQWKC3LW24Y565B52YJCQ, seg_pos={0 0}, upload=668]: Uploading piece... Oct 27 15:28:23.563479036 (45m40.912786323s) [noise=true , address=77.90.31.13:4626 , piece_id=GR3XECVFMZ5E3HDMKK2HLKJREMIOZGWIUI7J3NSK7QGB523G7YJQ, seg_pos={0 0}, upload=668]: Uploading piece... Oct 27 15:28:23.563711468 (45m40.913018754s) [noise=true , address=45.91.116.5:4298 , piece_id=ATXRIPPIX6BAHUXCENJVDJ25K2Z6UH3QYL4MTBWPZKP7LFGHBX6A, seg_pos={0 0}, upload=668]: Uploading piece... Oct 27 15:28:23.564137260 (45m40.913444546s) [noise=true , address=145.79.227.5:4223 , piece_id=FHAUWKUQAHGYR6ZJHKBCXT3ZPBG42MMJSADUGU7CIFUO3JPPNKKA, seg_pos={0 0}, upload=668]: Uploading piece... Oct 27 15:28:27.241369967 (45m44.590677243s) [upload=668]: Next returned nil segment Oct 27 15:28:27.241385446 (45m44.590692732s) [upload=668]: Waiting for error group managing segments... Oct 27 15:28:29.744963349 (45m47.094270636s) [noise=true , address=145.79.227.5:4222 , piece_id=B7W6PCNLLAPHXL7XOSJXEOVX6XZGNFDJWXR7MVYDGCONQCN55HYA, seg_pos={0 0}, upload=668]: Done uploading piece. err: <nil> Oct 27 15:28:29.821202504 (45m47.17050978s) [noise=true , address=145.79.227.21:4475 , piece_id=VDOFRCJKDQLXV4CBM4MA3UGK2PZMRJ3BW6H7PKJRZMZXW56Y5G4A, seg_pos={0 0}, upload=668]: Done uploading piece. err: <nil> Oct 27 15:28:32.126653324 (45m49.47596061s) [noise=true , address=77.90.31.18:4807 , piece_id=OPDBUFUCTPZN2HH2SORHJL5L4EX2VWXATEFEIIEN5QTBQQFRGSZA, seg_pos={0 0}, upload=668]: Done uploading piece. err: <nil> Oct 27 15:28:33.087272849 (45m50.436580125s) [noise=true , address=45.91.116.5:4298 , piece_id=ATXRIPPIX6BAHUXCENJVDJ25K2Z6UH3QYL4MTBWPZKP7LFGHBX6A, seg_pos={0 0}, upload=668]: Done uploading piece. err: <nil> Oct 27 15:28:38.581899084 (45m55.93120637s) [noise=true , address=145.79.227.26:4653 , piece_id=MXREE4NCFOGYEHVQNFHZYD5DLD4PHBJS64JYSLTXXJZG7AC3LH6Q, seg_pos={0 0}, upload=668]: Done uploading piece. err: <nil> Oct 27 15:28:39.053322174 (45m56.40262946s) [noise=true , address=145.79.227.24:4582 , piece_id=K7COJQ2UG3CTGF4JVO27PTGWY4E3XGGRQWKC3LW24Y565B52YJCQ, seg_pos={0 0}, upload=668]: Done uploading piece. err: <nil> Oct 27 15:28:39.717124420 (45m57.066431696s) [noise=true , address=145.79.227.20:4437 , piece_id=2EGSCSNCQJFXYT54PWEM33SP3RUYICBKKE7IDRQNMZ2ONPM6I5UA, seg_pos={0 0}, upload=668]: Done uploading piece. err: <nil> Oct 27 15:28:41.618415745 (45m58.967723031s) [noise=true , address=77.90.31.13:4626 , piece_id=GR3XECVFMZ5E3HDMKK2HLKJREMIOZGWIUI7J3NSK7QGB523G7YJQ, seg_pos={0 0}, upload=668]: Done uploading piece. err: <nil> Oct 27 15:28:42.068020890 (45m59.417328166s) [noise=true , address=145.79.227.5:4223 , piece_id=FHAUWKUQAHGYR6ZJHKBCXT3ZPBG42MMJSADUGU7CIFUO3JPPNKKA, seg_pos={0 0}, upload=668]: Done uploading piece. err: <nil> Oct 27 15:28:42.150893235 (45m59.500200521s) [noise=true , address=77.90.31.10:4518 , piece_id=76MPB5O5XOR54TOXYCUNK2RBQMWYTWADCBAZV2KQYJ5K7XXDVFMQ, seg_pos={0 0}, upload=668]: Done uploading piece. err: <nil> Oct 27 15:28:43.703285619 (46m1.052592905s) [noise=true , address=145.79.227.9:4366 , piece_id=5XLRE4OGUCQGZA5UXVNTNTFAMAGGQNHRR4MX23L3362K2DZLDREQ, seg_pos={0 0}, upload=668]: Done uploading piece. err: <nil> Oct 27 15:28:49.699904003 (46m7.049211289s) [noise=true , address=145.79.227.4:4186 , piece_id=6APC73J3JHGVX3ZNCN3UL5DLHXRYORPQT6AYPFPX6ZQLQIRF5TNA, seg_pos={0 0}, upload=668]: Done uploading piece. err: <nil> Oct 27 15:28:49.842045738 (46m7.191353014s) [noise=true , address=145.79.227.7:4294 , piece_id=QFLUZYLIGHO5E3ABEA5LFJPHAKDOF3J55Q4BRF5NXVKDNEHV5DBA, seg_pos={0 0}, upload=668]: Done uploading piece. err: <nil> Oct 27 15:28:49.842081656 (46m7.191388932s) [seg_pos={0 0}, upload=668]: Segment reached optimal threshold of 13 pieces. Oct 27 15:28:49.842296495 (46m7.191603781s) [noise=true , address=145.79.227.17:4971 , piece_id=NK55UPWQTKDWAAMPQTJPPOMZXH5VQGQ3QHJKRNMIRPFLFKQBZZIA, seg_pos={0 0}, upload=668]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo): protocol: write tcp 188.166.225.170:55424->145.79.227.17:4971: use of closed network connection; write tcp 188.166.225.170:55424->145.79.227.17:4971: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55424->145.79.227.17:4971: use of closed network connection Oct 27 15:28:49.842306875 (46m7.191614161s) [noise=true , address=145.79.227.17:4971 , piece_id=NK55UPWQTKDWAAMPQTJPPOMZXH5VQGQ3QHJKRNMIRPFLFKQBZZIA, seg_pos={0 0}, upload=668]: Long tail cancellation due to optimal threshold Oct 27 15:28:49.842335229 (46m7.191642535s) [noise=true , address=145.79.227.10:4401 , piece_id=SZNSLFKN44YVSUHRXZTF6BBVPYQODYVEBWQF3JHP6ZDA2A723WAA, seg_pos={0 0}, upload=668]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1): protocol: write tcp 188.166.225.170:50900->145.79.227.10:4401: use of closed network connection; write tcp 188.166.225.170:50900->145.79.227.10:4401: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50900->145.79.227.10:4401: use of closed network connection Oct 27 15:28:49.842355578 (46m7.191662874s) [noise=true , address=145.79.227.10:4401 , piece_id=SZNSLFKN44YVSUHRXZTF6BBVPYQODYVEBWQF3JHP6ZDA2A723WAA, seg_pos={0 0}, upload=668]: Long tail cancellation due to optimal threshold Oct 27 15:28:49.842403068 (46m7.191710354s) [noise=true , address=145.79.227.19:4079 , piece_id=QXNYHJS2SBTG2QRH7AKW7NZ6SJG7MFO5DHXCMBZHGT6PJTOZ526Q, seg_pos={0 0}, upload=668]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud): protocol: write tcp 188.166.225.170:37220->145.79.227.19:4079: use of closed network connection; write tcp 188.166.225.170:37220->145.79.227.19:4079: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37220->145.79.227.19:4079: use of closed network connection Oct 27 15:28:49.842411143 (46m7.191718419s) [noise=true , address=145.79.227.19:4079 , piece_id=QXNYHJS2SBTG2QRH7AKW7NZ6SJG7MFO5DHXCMBZHGT6PJTOZ526Q, seg_pos={0 0}, upload=668]: Long tail cancellation due to optimal threshold Oct 27 15:28:49.842428026 (46m7.191735312s) [seg_pos={0 0}, upload=668]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 27 15:28:49.842507998 (46m7.191815284s) [upload=668]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 27 15:28:49.842513478 (46m7.191820764s) [upload=668]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 27 15:28:50.170126807 (46m7.519434093s) [upload=668]: Done uploading segments... Oct 27 15:28:50.170148138 (46m7.519455424s) [upload=668]: Upload finished. err: <nil> Duration: 5.952665322s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 6.098168256s FAIL (2025-10-27 15:30:32 UTC) orbiter warning: context deadline exceeded Duration: 6.098195118s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 3/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=3) [........................X..............XX................X..] duration: 3.407915242s FAIL (2025-10-27 15:30:28 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.407942183s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)