Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 10.581161103s trace: https://jaeger.dev.storj.io/trace/8ca15e950587d46 FAIL (2025-12-14 13:40:54 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12YKLTah3swX14kL82eNogJxd8D8Hjfq7SPjDDQAapDnXmDctP1, Piece ID: GNFCQWHHQHCCHL7VNU6TYNEFJZFGR7SMXQO66UCSNORVK4XAHPGA) context deadline exceeded; piecestore: (Node ID: 125GxLsL5UtXkuJ1LFvF42N9iZkKtRK3rq63xdR25JQbtwoT1L5, Piece ID: AFT773CWP62ANAIFPUTDYQ35GSF6K2NFEAO7X4VKIT62CQD3NJEQ) context deadline exceeded ======LOGS====== Dec 14 13:22:35.112522281 (213h53m34.983605316s) [upload=187397]: Starting upload Dec 14 13:22:35.112587294 (213h53m34.983670329s) [upload=187397]: Done starting upload Dec 14 13:22:35.112673899 (213h53m34.983756934s) [upload=187397]: Uploading segments... Dec 14 13:22:35.112767347 (213h53m34.983850393s) [upload=187397]: Got next segment. Inline: false Dec 14 13:22:35.112800511 (213h53m34.983883556s) [upload=187397]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 14 13:22:35.112805821 (213h53m34.983888856s) [upload=187397]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 14 13:22:37.194903810 (213h53m37.065986845s) [seg_pos={0 0}, upload=187397]: Begin upload segment... Dec 14 13:22:37.195008499 (213h53m37.066091534s) [noise=true , address=145.79.227.42:4123 , piece_id=ES2R3XH3RQ2UAJTCWRLZBY3VHUZQVUA2EMCNBYBQSTR3N7NWU6EQ, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.195025751 (213h53m37.066108786s) [seg_pos={0 0}, upload=187397]: Done begin upload segment. Dec 14 13:22:37.195062162 (213h53m37.066145197s) [noise=true , address=145.79.227.44:4193 , piece_id=UMQOWDMKE3PCVS6DQUPMMRR3IYW7EUY3ABOGSFVJ3H7KGFRUZ7HA, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.195125401 (213h53m37.066208446s) [noise=true , address=145.79.227.45:4229 , piece_id=4IHXIUBBXYN6MZYQBRBRMJZ5HYOFNCLHKOVZKFVS6M37QMRE35OA, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.195764098 (213h53m37.066847133s) [noise=true , address=145.79.227.35:4482 , piece_id=RAUYG5T6DQIDOV5QM2OCE4GAPYIFBHUA3ZZR3Y6KN7ZX4OPMPEZA, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.196145995 (213h53m37.06722904s) [noise=true , address=145.79.227.47:4267 , piece_id=FOKTCGQ7VI4GDXRJ64LDLVSALAHAL7GRFI257S6X6G2Y4IGUNW6Q, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.196522081 (213h53m37.067605116s) [noise=true , address=145.79.227.36:4519 , piece_id=5X7PDP7FELCD6A5566SD7YBEUN4WELNB4IDV545B27TJ6DPEIIKA, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.196919738 (213h53m37.068002783s) [noise=true , address=145.79.227.40:4050 , piece_id=OFTY43B5JLDTXZRWXEHQZLUP6YJOL75CAL36EHSOQJR3GIHZOAJQ, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.197734449 (213h53m37.068817484s) [noise=true , address=145.79.227.28:4725 , piece_id=GNFCQWHHQHCCHL7VNU6TYNEFJZFGR7SMXQO66UCSNORVK4XAHPGA, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.197786599 (213h53m37.068869634s) [noise=true , address=145.79.227.43:4158 , piece_id=HDWGVVZAZI3R6KQFIUG74SWRF3G2USEMNKVNMKZDYISPRA74AUGQ, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.198256694 (213h53m37.069339739s) [noise=true , address=145.79.227.35:4481 , piece_id=LTU57UBRXVWPXJKKRNWYSIQHL7YYPNDMI7R5GARIP55AO7KS4TXQ, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.198675992 (213h53m37.069759027s) [noise=true , address=145.79.227.51:4411 , piece_id=V7ISMKJUJ6EDCLYOJIZBCBUUNDIWADZPZUUY7XH2F6BHVZILRXBQ, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.199111532 (213h53m37.070194567s) [noise=true , address=145.79.227.26:4655 , piece_id=NSYNQURKGVD7VTPMXJGF5RDIQKNUBLVKFYUE5CLPQ6IMIEJZQ4SQ, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.199658874 (213h53m37.07074191s) [noise=true , address=145.79.227.34:4905 , piece_id=GHVXBDDAG64LE2A5RRGK3POPOL3QSQGGI7UH3NIHEZTEG7A7GODA, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.200449229 (213h53m37.071532274s) [noise=true , address=145.79.227.38:4978 , piece_id=3AV4CLNBUBIXI73A3YSDKOPIMOROFBI73TVA3KNGDSBSE6KMYNEA, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.200877725 (213h53m37.07196077s) [noise=true , address=145.79.227.46:4601 , piece_id=AFT773CWP62ANAIFPUTDYQ35GSF6K2NFEAO7X4VKIT62CQD3NJEQ, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.207413279 (213h53m37.078496314s) [noise=true , address=145.79.227.31:4797 , piece_id=KRMTNOTWOWCIERSEUWBWZOIPDDUWYZKUBX74ZAIWV737CIKFVRLA, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:39.731884038 (213h53m39.602967073s) [upload=187397]: Next returned nil segment Dec 14 13:22:39.731901430 (213h53m39.602984475s) [upload=187397]: Waiting for error group managing segments... Dec 14 13:22:40.154962424 (213h53m40.026045469s) [noise=true , address=145.79.227.26:4655 , piece_id=NSYNQURKGVD7VTPMXJGF5RDIQKNUBLVKFYUE5CLPQ6IMIEJZQ4SQ, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.159418800 (213h53m40.030501835s) [noise=true , address=145.79.227.40:4050 , piece_id=OFTY43B5JLDTXZRWXEHQZLUP6YJOL75CAL36EHSOQJR3GIHZOAJQ, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.175663783 (213h53m40.046746829s) [noise=true , address=145.79.227.43:4158 , piece_id=HDWGVVZAZI3R6KQFIUG74SWRF3G2USEMNKVNMKZDYISPRA74AUGQ, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.175938396 (213h53m40.047021432s) [noise=true , address=145.79.227.35:4482 , piece_id=RAUYG5T6DQIDOV5QM2OCE4GAPYIFBHUA3ZZR3Y6KN7ZX4OPMPEZA, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.177599980 (213h53m40.048683025s) [noise=true , address=145.79.227.31:4797 , piece_id=KRMTNOTWOWCIERSEUWBWZOIPDDUWYZKUBX74ZAIWV737CIKFVRLA, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.181351754 (213h53m40.052434789s) [noise=true , address=145.79.227.51:4411 , piece_id=V7ISMKJUJ6EDCLYOJIZBCBUUNDIWADZPZUUY7XH2F6BHVZILRXBQ, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.181709967 (213h53m40.052793002s) [noise=true , address=145.79.227.46:4601 , piece_id=AFT773CWP62ANAIFPUTDYQ35GSF6K2NFEAO7X4VKIT62CQD3NJEQ, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.183204752 (213h53m40.054287787s) [noise=true , address=145.79.227.28:4725 , piece_id=GNFCQWHHQHCCHL7VNU6TYNEFJZFGR7SMXQO66UCSNORVK4XAHPGA, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.183813983 (213h53m40.054897028s) [noise=true , address=145.79.227.45:4229 , piece_id=4IHXIUBBXYN6MZYQBRBRMJZ5HYOFNCLHKOVZKFVS6M37QMRE35OA, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.186380270 (213h53m40.057463315s) [noise=true , address=145.79.227.38:4978 , piece_id=3AV4CLNBUBIXI73A3YSDKOPIMOROFBI73TVA3KNGDSBSE6KMYNEA, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.198000036 (213h53m40.069083071s) [noise=true , address=145.79.227.36:4519 , piece_id=5X7PDP7FELCD6A5566SD7YBEUN4WELNB4IDV545B27TJ6DPEIIKA, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.201922084 (213h53m40.07300513s) [noise=true , address=145.79.227.47:4267 , piece_id=FOKTCGQ7VI4GDXRJ64LDLVSALAHAL7GRFI257S6X6G2Y4IGUNW6Q, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.217629133 (213h53m40.088712178s) [noise=true , address=145.79.227.34:4905 , piece_id=GHVXBDDAG64LE2A5RRGK3POPOL3QSQGGI7UH3NIHEZTEG7A7GODA, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.217662106 (213h53m40.088745151s) [seg_pos={0 0}, upload=187397]: Segment reached optimal threshold of 13 pieces. Dec 14 13:22:40.218130388 (213h53m40.089213423s) [noise=true , address=145.79.227.44:4193 , piece_id=UMQOWDMKE3PCVS6DQUPMMRR3IYW7EUY3ABOGSFVJ3H7KGFRUZ7HA, seg_pos={0 0}, upload=187397]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8): protocol: expected piece hash; context canceled; EOF Dec 14 13:22:40.218148793 (213h53m40.089231828s) [noise=true , address=145.79.227.44:4193 , piece_id=UMQOWDMKE3PCVS6DQUPMMRR3IYW7EUY3ABOGSFVJ3H7KGFRUZ7HA, seg_pos={0 0}, upload=187397]: Long tail cancellation due to optimal threshold Dec 14 13:22:40.218265846 (213h53m40.089348891s) [noise=true , address=145.79.227.35:4481 , piece_id=LTU57UBRXVWPXJKKRNWYSIQHL7YYPNDMI7R5GARIP55AO7KS4TXQ, seg_pos={0 0}, upload=187397]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bRK8WWmAnWUwzCfHu8JtZgrruYSJCvhnsYTKcaLccJcv5RQjp): protocol: expected piece hash; context canceled; EOF Dec 14 13:22:40.218275184 (213h53m40.089358239s) [noise=true , address=145.79.227.35:4481 , piece_id=LTU57UBRXVWPXJKKRNWYSIQHL7YYPNDMI7R5GARIP55AO7KS4TXQ, seg_pos={0 0}, upload=187397]: Long tail cancellation due to optimal threshold Dec 14 13:22:40.218315581 (213h53m40.089398616s) [noise=true , address=145.79.227.42:4123 , piece_id=ES2R3XH3RQ2UAJTCWRLZBY3VHUZQVUA2EMCNBYBQSTR3N7NWU6EQ, seg_pos={0 0}, upload=187397]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM): protocol: expected piece hash; context canceled; EOF Dec 14 13:22:40.218330358 (213h53m40.089413393s) [noise=true , address=145.79.227.42:4123 , piece_id=ES2R3XH3RQ2UAJTCWRLZBY3VHUZQVUA2EMCNBYBQSTR3N7NWU6EQ, seg_pos={0 0}, upload=187397]: Long tail cancellation due to optimal threshold Dec 14 13:22:40.218347892 (213h53m40.089430937s) [seg_pos={0 0}, upload=187397]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 14 13:22:40.218452501 (213h53m40.089535556s) [upload=187397]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 14 13:22:40.218479914 (213h53m40.089562969s) [upload=187397]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 14 13:22:43.384827291 (213h53m43.255910336s) [upload=187397]: Done uploading segments... Dec 14 13:22:43.384857198 (213h53m43.255940233s) [upload=187397]: Upload finished. err: <nil> Duration: 10.58119036s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [........................................X.......X...X..X....] duration: 3.600960767s trace: https://jaeger.dev.storj.io/trace/3891d01bf25a7e1e FAIL (2025-12-14 13:40:48 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.600987369s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [......X...................................X................X] duration: 30.001000207s trace: https://jaeger.dev.storj.io/trace/6d3d9ecb5c3a7830 FAIL (2025-12-14 13:40:45 UTC) orbiter warning: context deadline exceeded Duration: 30.00102247s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 8.802200284s trace: https://jaeger.dev.storj.io/trace/8ca15e950587d46 FAIL (2025-12-14 13:40:35 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12YKLTah3swX14kL82eNogJxd8D8Hjfq7SPjDDQAapDnXmDctP1, Piece ID: GNFCQWHHQHCCHL7VNU6TYNEFJZFGR7SMXQO66UCSNORVK4XAHPGA) context deadline exceeded; piecestore: (Node ID: 125GxLsL5UtXkuJ1LFvF42N9iZkKtRK3rq63xdR25JQbtwoT1L5, Piece ID: AFT773CWP62ANAIFPUTDYQ35GSF6K2NFEAO7X4VKIT62CQD3NJEQ) context deadline exceeded ======LOGS====== Dec 14 13:22:35.112522281 (213h53m34.983605316s) [upload=187397]: Starting upload Dec 14 13:22:35.112587294 (213h53m34.983670329s) [upload=187397]: Done starting upload Dec 14 13:22:35.112673899 (213h53m34.983756934s) [upload=187397]: Uploading segments... Dec 14 13:22:35.112767347 (213h53m34.983850393s) [upload=187397]: Got next segment. Inline: false Dec 14 13:22:35.112800511 (213h53m34.983883556s) [upload=187397]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 14 13:22:35.112805821 (213h53m34.983888856s) [upload=187397]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 14 13:22:37.194903810 (213h53m37.065986845s) [seg_pos={0 0}, upload=187397]: Begin upload segment... Dec 14 13:22:37.195008499 (213h53m37.066091534s) [noise=true , address=145.79.227.42:4123 , piece_id=ES2R3XH3RQ2UAJTCWRLZBY3VHUZQVUA2EMCNBYBQSTR3N7NWU6EQ, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.195025751 (213h53m37.066108786s) [seg_pos={0 0}, upload=187397]: Done begin upload segment. Dec 14 13:22:37.195062162 (213h53m37.066145197s) [noise=true , address=145.79.227.44:4193 , piece_id=UMQOWDMKE3PCVS6DQUPMMRR3IYW7EUY3ABOGSFVJ3H7KGFRUZ7HA, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.195125401 (213h53m37.066208446s) [noise=true , address=145.79.227.45:4229 , piece_id=4IHXIUBBXYN6MZYQBRBRMJZ5HYOFNCLHKOVZKFVS6M37QMRE35OA, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.195764098 (213h53m37.066847133s) [noise=true , address=145.79.227.35:4482 , piece_id=RAUYG5T6DQIDOV5QM2OCE4GAPYIFBHUA3ZZR3Y6KN7ZX4OPMPEZA, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.196145995 (213h53m37.06722904s) [noise=true , address=145.79.227.47:4267 , piece_id=FOKTCGQ7VI4GDXRJ64LDLVSALAHAL7GRFI257S6X6G2Y4IGUNW6Q, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.196522081 (213h53m37.067605116s) [noise=true , address=145.79.227.36:4519 , piece_id=5X7PDP7FELCD6A5566SD7YBEUN4WELNB4IDV545B27TJ6DPEIIKA, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.196919738 (213h53m37.068002783s) [noise=true , address=145.79.227.40:4050 , piece_id=OFTY43B5JLDTXZRWXEHQZLUP6YJOL75CAL36EHSOQJR3GIHZOAJQ, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.197734449 (213h53m37.068817484s) [noise=true , address=145.79.227.28:4725 , piece_id=GNFCQWHHQHCCHL7VNU6TYNEFJZFGR7SMXQO66UCSNORVK4XAHPGA, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.197786599 (213h53m37.068869634s) [noise=true , address=145.79.227.43:4158 , piece_id=HDWGVVZAZI3R6KQFIUG74SWRF3G2USEMNKVNMKZDYISPRA74AUGQ, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.198256694 (213h53m37.069339739s) [noise=true , address=145.79.227.35:4481 , piece_id=LTU57UBRXVWPXJKKRNWYSIQHL7YYPNDMI7R5GARIP55AO7KS4TXQ, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.198675992 (213h53m37.069759027s) [noise=true , address=145.79.227.51:4411 , piece_id=V7ISMKJUJ6EDCLYOJIZBCBUUNDIWADZPZUUY7XH2F6BHVZILRXBQ, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.199111532 (213h53m37.070194567s) [noise=true , address=145.79.227.26:4655 , piece_id=NSYNQURKGVD7VTPMXJGF5RDIQKNUBLVKFYUE5CLPQ6IMIEJZQ4SQ, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.199658874 (213h53m37.07074191s) [noise=true , address=145.79.227.34:4905 , piece_id=GHVXBDDAG64LE2A5RRGK3POPOL3QSQGGI7UH3NIHEZTEG7A7GODA, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.200449229 (213h53m37.071532274s) [noise=true , address=145.79.227.38:4978 , piece_id=3AV4CLNBUBIXI73A3YSDKOPIMOROFBI73TVA3KNGDSBSE6KMYNEA, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.200877725 (213h53m37.07196077s) [noise=true , address=145.79.227.46:4601 , piece_id=AFT773CWP62ANAIFPUTDYQ35GSF6K2NFEAO7X4VKIT62CQD3NJEQ, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.207413279 (213h53m37.078496314s) [noise=true , address=145.79.227.31:4797 , piece_id=KRMTNOTWOWCIERSEUWBWZOIPDDUWYZKUBX74ZAIWV737CIKFVRLA, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:39.731884038 (213h53m39.602967073s) [upload=187397]: Next returned nil segment Dec 14 13:22:39.731901430 (213h53m39.602984475s) [upload=187397]: Waiting for error group managing segments... Dec 14 13:22:40.154962424 (213h53m40.026045469s) [noise=true , address=145.79.227.26:4655 , piece_id=NSYNQURKGVD7VTPMXJGF5RDIQKNUBLVKFYUE5CLPQ6IMIEJZQ4SQ, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.159418800 (213h53m40.030501835s) [noise=true , address=145.79.227.40:4050 , piece_id=OFTY43B5JLDTXZRWXEHQZLUP6YJOL75CAL36EHSOQJR3GIHZOAJQ, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.175663783 (213h53m40.046746829s) [noise=true , address=145.79.227.43:4158 , piece_id=HDWGVVZAZI3R6KQFIUG74SWRF3G2USEMNKVNMKZDYISPRA74AUGQ, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.175938396 (213h53m40.047021432s) [noise=true , address=145.79.227.35:4482 , piece_id=RAUYG5T6DQIDOV5QM2OCE4GAPYIFBHUA3ZZR3Y6KN7ZX4OPMPEZA, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.177599980 (213h53m40.048683025s) [noise=true , address=145.79.227.31:4797 , piece_id=KRMTNOTWOWCIERSEUWBWZOIPDDUWYZKUBX74ZAIWV737CIKFVRLA, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.181351754 (213h53m40.052434789s) [noise=true , address=145.79.227.51:4411 , piece_id=V7ISMKJUJ6EDCLYOJIZBCBUUNDIWADZPZUUY7XH2F6BHVZILRXBQ, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.181709967 (213h53m40.052793002s) [noise=true , address=145.79.227.46:4601 , piece_id=AFT773CWP62ANAIFPUTDYQ35GSF6K2NFEAO7X4VKIT62CQD3NJEQ, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.183204752 (213h53m40.054287787s) [noise=true , address=145.79.227.28:4725 , piece_id=GNFCQWHHQHCCHL7VNU6TYNEFJZFGR7SMXQO66UCSNORVK4XAHPGA, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.183813983 (213h53m40.054897028s) [noise=true , address=145.79.227.45:4229 , piece_id=4IHXIUBBXYN6MZYQBRBRMJZ5HYOFNCLHKOVZKFVS6M37QMRE35OA, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.186380270 (213h53m40.057463315s) [noise=true , address=145.79.227.38:4978 , piece_id=3AV4CLNBUBIXI73A3YSDKOPIMOROFBI73TVA3KNGDSBSE6KMYNEA, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.198000036 (213h53m40.069083071s) [noise=true , address=145.79.227.36:4519 , piece_id=5X7PDP7FELCD6A5566SD7YBEUN4WELNB4IDV545B27TJ6DPEIIKA, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.201922084 (213h53m40.07300513s) [noise=true , address=145.79.227.47:4267 , piece_id=FOKTCGQ7VI4GDXRJ64LDLVSALAHAL7GRFI257S6X6G2Y4IGUNW6Q, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.217629133 (213h53m40.088712178s) [noise=true , address=145.79.227.34:4905 , piece_id=GHVXBDDAG64LE2A5RRGK3POPOL3QSQGGI7UH3NIHEZTEG7A7GODA, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.217662106 (213h53m40.088745151s) [seg_pos={0 0}, upload=187397]: Segment reached optimal threshold of 13 pieces. Dec 14 13:22:40.218130388 (213h53m40.089213423s) [noise=true , address=145.79.227.44:4193 , piece_id=UMQOWDMKE3PCVS6DQUPMMRR3IYW7EUY3ABOGSFVJ3H7KGFRUZ7HA, seg_pos={0 0}, upload=187397]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8): protocol: expected piece hash; context canceled; EOF Dec 14 13:22:40.218148793 (213h53m40.089231828s) [noise=true , address=145.79.227.44:4193 , piece_id=UMQOWDMKE3PCVS6DQUPMMRR3IYW7EUY3ABOGSFVJ3H7KGFRUZ7HA, seg_pos={0 0}, upload=187397]: Long tail cancellation due to optimal threshold Dec 14 13:22:40.218265846 (213h53m40.089348891s) [noise=true , address=145.79.227.35:4481 , piece_id=LTU57UBRXVWPXJKKRNWYSIQHL7YYPNDMI7R5GARIP55AO7KS4TXQ, seg_pos={0 0}, upload=187397]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bRK8WWmAnWUwzCfHu8JtZgrruYSJCvhnsYTKcaLccJcv5RQjp): protocol: expected piece hash; context canceled; EOF Dec 14 13:22:40.218275184 (213h53m40.089358239s) [noise=true , address=145.79.227.35:4481 , piece_id=LTU57UBRXVWPXJKKRNWYSIQHL7YYPNDMI7R5GARIP55AO7KS4TXQ, seg_pos={0 0}, upload=187397]: Long tail cancellation due to optimal threshold Dec 14 13:22:40.218315581 (213h53m40.089398616s) [noise=true , address=145.79.227.42:4123 , piece_id=ES2R3XH3RQ2UAJTCWRLZBY3VHUZQVUA2EMCNBYBQSTR3N7NWU6EQ, seg_pos={0 0}, upload=187397]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM): protocol: expected piece hash; context canceled; EOF Dec 14 13:22:40.218330358 (213h53m40.089413393s) [noise=true , address=145.79.227.42:4123 , piece_id=ES2R3XH3RQ2UAJTCWRLZBY3VHUZQVUA2EMCNBYBQSTR3N7NWU6EQ, seg_pos={0 0}, upload=187397]: Long tail cancellation due to optimal threshold Dec 14 13:22:40.218347892 (213h53m40.089430937s) [seg_pos={0 0}, upload=187397]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 14 13:22:40.218452501 (213h53m40.089535556s) [upload=187397]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 14 13:22:40.218479914 (213h53m40.089562969s) [upload=187397]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 14 13:22:43.384827291 (213h53m43.255910336s) [upload=187397]: Done uploading segments... Dec 14 13:22:43.384857198 (213h53m43.255940233s) [upload=187397]: Upload finished. err: <nil> Duration: 8.802227435s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [.........................................X.......X...X..X...] duration: 5.001256703s trace: https://jaeger.dev.storj.io/trace/3891d01bf25a7e1e FAIL (2025-12-14 13:40:10 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.00128127s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [..........................................X.......X...X..X..] duration: 5.001385428s trace: https://jaeger.dev.storj.io/trace/3891d01bf25a7e1e FAIL (2025-12-14 13:39:49 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001406649s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......X...................................X................] duration: 19.13233163s trace: https://jaeger.dev.storj.io/trace/2f31d4ad7317aa95 FAIL (2025-12-14 13:39:38 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 19.13236325s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 9.689244347s trace: https://jaeger.dev.storj.io/trace/8ca15e950587d46 FAIL (2025-12-14 13:39:28 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12YKLTah3swX14kL82eNogJxd8D8Hjfq7SPjDDQAapDnXmDctP1, Piece ID: GNFCQWHHQHCCHL7VNU6TYNEFJZFGR7SMXQO66UCSNORVK4XAHPGA) context deadline exceeded; piecestore: (Node ID: 125GxLsL5UtXkuJ1LFvF42N9iZkKtRK3rq63xdR25JQbtwoT1L5, Piece ID: AFT773CWP62ANAIFPUTDYQ35GSF6K2NFEAO7X4VKIT62CQD3NJEQ) context deadline exceeded ======LOGS====== Dec 14 13:22:35.112522281 (213h53m34.983605316s) [upload=187397]: Starting upload Dec 14 13:22:35.112587294 (213h53m34.983670329s) [upload=187397]: Done starting upload Dec 14 13:22:35.112673899 (213h53m34.983756934s) [upload=187397]: Uploading segments... Dec 14 13:22:35.112767347 (213h53m34.983850393s) [upload=187397]: Got next segment. Inline: false Dec 14 13:22:35.112800511 (213h53m34.983883556s) [upload=187397]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 14 13:22:35.112805821 (213h53m34.983888856s) [upload=187397]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 14 13:22:37.194903810 (213h53m37.065986845s) [seg_pos={0 0}, upload=187397]: Begin upload segment... Dec 14 13:22:37.195008499 (213h53m37.066091534s) [noise=true , address=145.79.227.42:4123 , piece_id=ES2R3XH3RQ2UAJTCWRLZBY3VHUZQVUA2EMCNBYBQSTR3N7NWU6EQ, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.195025751 (213h53m37.066108786s) [seg_pos={0 0}, upload=187397]: Done begin upload segment. Dec 14 13:22:37.195062162 (213h53m37.066145197s) [noise=true , address=145.79.227.44:4193 , piece_id=UMQOWDMKE3PCVS6DQUPMMRR3IYW7EUY3ABOGSFVJ3H7KGFRUZ7HA, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.195125401 (213h53m37.066208446s) [noise=true , address=145.79.227.45:4229 , piece_id=4IHXIUBBXYN6MZYQBRBRMJZ5HYOFNCLHKOVZKFVS6M37QMRE35OA, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.195764098 (213h53m37.066847133s) [noise=true , address=145.79.227.35:4482 , piece_id=RAUYG5T6DQIDOV5QM2OCE4GAPYIFBHUA3ZZR3Y6KN7ZX4OPMPEZA, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.196145995 (213h53m37.06722904s) [noise=true , address=145.79.227.47:4267 , piece_id=FOKTCGQ7VI4GDXRJ64LDLVSALAHAL7GRFI257S6X6G2Y4IGUNW6Q, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.196522081 (213h53m37.067605116s) [noise=true , address=145.79.227.36:4519 , piece_id=5X7PDP7FELCD6A5566SD7YBEUN4WELNB4IDV545B27TJ6DPEIIKA, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.196919738 (213h53m37.068002783s) [noise=true , address=145.79.227.40:4050 , piece_id=OFTY43B5JLDTXZRWXEHQZLUP6YJOL75CAL36EHSOQJR3GIHZOAJQ, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.197734449 (213h53m37.068817484s) [noise=true , address=145.79.227.28:4725 , piece_id=GNFCQWHHQHCCHL7VNU6TYNEFJZFGR7SMXQO66UCSNORVK4XAHPGA, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.197786599 (213h53m37.068869634s) [noise=true , address=145.79.227.43:4158 , piece_id=HDWGVVZAZI3R6KQFIUG74SWRF3G2USEMNKVNMKZDYISPRA74AUGQ, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.198256694 (213h53m37.069339739s) [noise=true , address=145.79.227.35:4481 , piece_id=LTU57UBRXVWPXJKKRNWYSIQHL7YYPNDMI7R5GARIP55AO7KS4TXQ, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.198675992 (213h53m37.069759027s) [noise=true , address=145.79.227.51:4411 , piece_id=V7ISMKJUJ6EDCLYOJIZBCBUUNDIWADZPZUUY7XH2F6BHVZILRXBQ, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.199111532 (213h53m37.070194567s) [noise=true , address=145.79.227.26:4655 , piece_id=NSYNQURKGVD7VTPMXJGF5RDIQKNUBLVKFYUE5CLPQ6IMIEJZQ4SQ, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.199658874 (213h53m37.07074191s) [noise=true , address=145.79.227.34:4905 , piece_id=GHVXBDDAG64LE2A5RRGK3POPOL3QSQGGI7UH3NIHEZTEG7A7GODA, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.200449229 (213h53m37.071532274s) [noise=true , address=145.79.227.38:4978 , piece_id=3AV4CLNBUBIXI73A3YSDKOPIMOROFBI73TVA3KNGDSBSE6KMYNEA, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.200877725 (213h53m37.07196077s) [noise=true , address=145.79.227.46:4601 , piece_id=AFT773CWP62ANAIFPUTDYQ35GSF6K2NFEAO7X4VKIT62CQD3NJEQ, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.207413279 (213h53m37.078496314s) [noise=true , address=145.79.227.31:4797 , piece_id=KRMTNOTWOWCIERSEUWBWZOIPDDUWYZKUBX74ZAIWV737CIKFVRLA, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:39.731884038 (213h53m39.602967073s) [upload=187397]: Next returned nil segment Dec 14 13:22:39.731901430 (213h53m39.602984475s) [upload=187397]: Waiting for error group managing segments... Dec 14 13:22:40.154962424 (213h53m40.026045469s) [noise=true , address=145.79.227.26:4655 , piece_id=NSYNQURKGVD7VTPMXJGF5RDIQKNUBLVKFYUE5CLPQ6IMIEJZQ4SQ, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.159418800 (213h53m40.030501835s) [noise=true , address=145.79.227.40:4050 , piece_id=OFTY43B5JLDTXZRWXEHQZLUP6YJOL75CAL36EHSOQJR3GIHZOAJQ, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.175663783 (213h53m40.046746829s) [noise=true , address=145.79.227.43:4158 , piece_id=HDWGVVZAZI3R6KQFIUG74SWRF3G2USEMNKVNMKZDYISPRA74AUGQ, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.175938396 (213h53m40.047021432s) [noise=true , address=145.79.227.35:4482 , piece_id=RAUYG5T6DQIDOV5QM2OCE4GAPYIFBHUA3ZZR3Y6KN7ZX4OPMPEZA, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.177599980 (213h53m40.048683025s) [noise=true , address=145.79.227.31:4797 , piece_id=KRMTNOTWOWCIERSEUWBWZOIPDDUWYZKUBX74ZAIWV737CIKFVRLA, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.181351754 (213h53m40.052434789s) [noise=true , address=145.79.227.51:4411 , piece_id=V7ISMKJUJ6EDCLYOJIZBCBUUNDIWADZPZUUY7XH2F6BHVZILRXBQ, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.181709967 (213h53m40.052793002s) [noise=true , address=145.79.227.46:4601 , piece_id=AFT773CWP62ANAIFPUTDYQ35GSF6K2NFEAO7X4VKIT62CQD3NJEQ, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.183204752 (213h53m40.054287787s) [noise=true , address=145.79.227.28:4725 , piece_id=GNFCQWHHQHCCHL7VNU6TYNEFJZFGR7SMXQO66UCSNORVK4XAHPGA, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.183813983 (213h53m40.054897028s) [noise=true , address=145.79.227.45:4229 , piece_id=4IHXIUBBXYN6MZYQBRBRMJZ5HYOFNCLHKOVZKFVS6M37QMRE35OA, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.186380270 (213h53m40.057463315s) [noise=true , address=145.79.227.38:4978 , piece_id=3AV4CLNBUBIXI73A3YSDKOPIMOROFBI73TVA3KNGDSBSE6KMYNEA, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.198000036 (213h53m40.069083071s) [noise=true , address=145.79.227.36:4519 , piece_id=5X7PDP7FELCD6A5566SD7YBEUN4WELNB4IDV545B27TJ6DPEIIKA, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.201922084 (213h53m40.07300513s) [noise=true , address=145.79.227.47:4267 , piece_id=FOKTCGQ7VI4GDXRJ64LDLVSALAHAL7GRFI257S6X6G2Y4IGUNW6Q, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.217629133 (213h53m40.088712178s) [noise=true , address=145.79.227.34:4905 , piece_id=GHVXBDDAG64LE2A5RRGK3POPOL3QSQGGI7UH3NIHEZTEG7A7GODA, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.217662106 (213h53m40.088745151s) [seg_pos={0 0}, upload=187397]: Segment reached optimal threshold of 13 pieces. Dec 14 13:22:40.218130388 (213h53m40.089213423s) [noise=true , address=145.79.227.44:4193 , piece_id=UMQOWDMKE3PCVS6DQUPMMRR3IYW7EUY3ABOGSFVJ3H7KGFRUZ7HA, seg_pos={0 0}, upload=187397]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8): protocol: expected piece hash; context canceled; EOF Dec 14 13:22:40.218148793 (213h53m40.089231828s) [noise=true , address=145.79.227.44:4193 , piece_id=UMQOWDMKE3PCVS6DQUPMMRR3IYW7EUY3ABOGSFVJ3H7KGFRUZ7HA, seg_pos={0 0}, upload=187397]: Long tail cancellation due to optimal threshold Dec 14 13:22:40.218265846 (213h53m40.089348891s) [noise=true , address=145.79.227.35:4481 , piece_id=LTU57UBRXVWPXJKKRNWYSIQHL7YYPNDMI7R5GARIP55AO7KS4TXQ, seg_pos={0 0}, upload=187397]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bRK8WWmAnWUwzCfHu8JtZgrruYSJCvhnsYTKcaLccJcv5RQjp): protocol: expected piece hash; context canceled; EOF Dec 14 13:22:40.218275184 (213h53m40.089358239s) [noise=true , address=145.79.227.35:4481 , piece_id=LTU57UBRXVWPXJKKRNWYSIQHL7YYPNDMI7R5GARIP55AO7KS4TXQ, seg_pos={0 0}, upload=187397]: Long tail cancellation due to optimal threshold Dec 14 13:22:40.218315581 (213h53m40.089398616s) [noise=true , address=145.79.227.42:4123 , piece_id=ES2R3XH3RQ2UAJTCWRLZBY3VHUZQVUA2EMCNBYBQSTR3N7NWU6EQ, seg_pos={0 0}, upload=187397]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM): protocol: expected piece hash; context canceled; EOF Dec 14 13:22:40.218330358 (213h53m40.089413393s) [noise=true , address=145.79.227.42:4123 , piece_id=ES2R3XH3RQ2UAJTCWRLZBY3VHUZQVUA2EMCNBYBQSTR3N7NWU6EQ, seg_pos={0 0}, upload=187397]: Long tail cancellation due to optimal threshold Dec 14 13:22:40.218347892 (213h53m40.089430937s) [seg_pos={0 0}, upload=187397]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 14 13:22:40.218452501 (213h53m40.089535556s) [upload=187397]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 14 13:22:40.218479914 (213h53m40.089562969s) [upload=187397]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 14 13:22:43.384827291 (213h53m43.255910336s) [upload=187397]: Done uploading segments... Dec 14 13:22:43.384857198 (213h53m43.255940233s) [upload=187397]: Upload finished. err: <nil> Duration: 9.689266498s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [...........................................X.......X...X..X.] duration: 5.000717278s trace: https://jaeger.dev.storj.io/trace/3891d01bf25a7e1e FAIL (2025-12-14 13:39:10 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000743949s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 8.850877809s trace: https://jaeger.dev.storj.io/trace/8ca15e950587d46 FAIL (2025-12-14 13:39:09 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12YKLTah3swX14kL82eNogJxd8D8Hjfq7SPjDDQAapDnXmDctP1, Piece ID: GNFCQWHHQHCCHL7VNU6TYNEFJZFGR7SMXQO66UCSNORVK4XAHPGA) context deadline exceeded; piecestore: (Node ID: 125GxLsL5UtXkuJ1LFvF42N9iZkKtRK3rq63xdR25JQbtwoT1L5, Piece ID: AFT773CWP62ANAIFPUTDYQ35GSF6K2NFEAO7X4VKIT62CQD3NJEQ) context deadline exceeded ======LOGS====== Dec 14 13:22:35.112522281 (213h53m34.983605316s) [upload=187397]: Starting upload Dec 14 13:22:35.112587294 (213h53m34.983670329s) [upload=187397]: Done starting upload Dec 14 13:22:35.112673899 (213h53m34.983756934s) [upload=187397]: Uploading segments... Dec 14 13:22:35.112767347 (213h53m34.983850393s) [upload=187397]: Got next segment. Inline: false Dec 14 13:22:35.112800511 (213h53m34.983883556s) [upload=187397]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 14 13:22:35.112805821 (213h53m34.983888856s) [upload=187397]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 14 13:22:37.194903810 (213h53m37.065986845s) [seg_pos={0 0}, upload=187397]: Begin upload segment... Dec 14 13:22:37.195008499 (213h53m37.066091534s) [noise=true , address=145.79.227.42:4123 , piece_id=ES2R3XH3RQ2UAJTCWRLZBY3VHUZQVUA2EMCNBYBQSTR3N7NWU6EQ, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.195025751 (213h53m37.066108786s) [seg_pos={0 0}, upload=187397]: Done begin upload segment. Dec 14 13:22:37.195062162 (213h53m37.066145197s) [noise=true , address=145.79.227.44:4193 , piece_id=UMQOWDMKE3PCVS6DQUPMMRR3IYW7EUY3ABOGSFVJ3H7KGFRUZ7HA, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.195125401 (213h53m37.066208446s) [noise=true , address=145.79.227.45:4229 , piece_id=4IHXIUBBXYN6MZYQBRBRMJZ5HYOFNCLHKOVZKFVS6M37QMRE35OA, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.195764098 (213h53m37.066847133s) [noise=true , address=145.79.227.35:4482 , piece_id=RAUYG5T6DQIDOV5QM2OCE4GAPYIFBHUA3ZZR3Y6KN7ZX4OPMPEZA, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.196145995 (213h53m37.06722904s) [noise=true , address=145.79.227.47:4267 , piece_id=FOKTCGQ7VI4GDXRJ64LDLVSALAHAL7GRFI257S6X6G2Y4IGUNW6Q, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.196522081 (213h53m37.067605116s) [noise=true , address=145.79.227.36:4519 , piece_id=5X7PDP7FELCD6A5566SD7YBEUN4WELNB4IDV545B27TJ6DPEIIKA, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.196919738 (213h53m37.068002783s) [noise=true , address=145.79.227.40:4050 , piece_id=OFTY43B5JLDTXZRWXEHQZLUP6YJOL75CAL36EHSOQJR3GIHZOAJQ, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.197734449 (213h53m37.068817484s) [noise=true , address=145.79.227.28:4725 , piece_id=GNFCQWHHQHCCHL7VNU6TYNEFJZFGR7SMXQO66UCSNORVK4XAHPGA, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.197786599 (213h53m37.068869634s) [noise=true , address=145.79.227.43:4158 , piece_id=HDWGVVZAZI3R6KQFIUG74SWRF3G2USEMNKVNMKZDYISPRA74AUGQ, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.198256694 (213h53m37.069339739s) [noise=true , address=145.79.227.35:4481 , piece_id=LTU57UBRXVWPXJKKRNWYSIQHL7YYPNDMI7R5GARIP55AO7KS4TXQ, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.198675992 (213h53m37.069759027s) [noise=true , address=145.79.227.51:4411 , piece_id=V7ISMKJUJ6EDCLYOJIZBCBUUNDIWADZPZUUY7XH2F6BHVZILRXBQ, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.199111532 (213h53m37.070194567s) [noise=true , address=145.79.227.26:4655 , piece_id=NSYNQURKGVD7VTPMXJGF5RDIQKNUBLVKFYUE5CLPQ6IMIEJZQ4SQ, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.199658874 (213h53m37.07074191s) [noise=true , address=145.79.227.34:4905 , piece_id=GHVXBDDAG64LE2A5RRGK3POPOL3QSQGGI7UH3NIHEZTEG7A7GODA, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.200449229 (213h53m37.071532274s) [noise=true , address=145.79.227.38:4978 , piece_id=3AV4CLNBUBIXI73A3YSDKOPIMOROFBI73TVA3KNGDSBSE6KMYNEA, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.200877725 (213h53m37.07196077s) [noise=true , address=145.79.227.46:4601 , piece_id=AFT773CWP62ANAIFPUTDYQ35GSF6K2NFEAO7X4VKIT62CQD3NJEQ, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.207413279 (213h53m37.078496314s) [noise=true , address=145.79.227.31:4797 , piece_id=KRMTNOTWOWCIERSEUWBWZOIPDDUWYZKUBX74ZAIWV737CIKFVRLA, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:39.731884038 (213h53m39.602967073s) [upload=187397]: Next returned nil segment Dec 14 13:22:39.731901430 (213h53m39.602984475s) [upload=187397]: Waiting for error group managing segments... Dec 14 13:22:40.154962424 (213h53m40.026045469s) [noise=true , address=145.79.227.26:4655 , piece_id=NSYNQURKGVD7VTPMXJGF5RDIQKNUBLVKFYUE5CLPQ6IMIEJZQ4SQ, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.159418800 (213h53m40.030501835s) [noise=true , address=145.79.227.40:4050 , piece_id=OFTY43B5JLDTXZRWXEHQZLUP6YJOL75CAL36EHSOQJR3GIHZOAJQ, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.175663783 (213h53m40.046746829s) [noise=true , address=145.79.227.43:4158 , piece_id=HDWGVVZAZI3R6KQFIUG74SWRF3G2USEMNKVNMKZDYISPRA74AUGQ, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.175938396 (213h53m40.047021432s) [noise=true , address=145.79.227.35:4482 , piece_id=RAUYG5T6DQIDOV5QM2OCE4GAPYIFBHUA3ZZR3Y6KN7ZX4OPMPEZA, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.177599980 (213h53m40.048683025s) [noise=true , address=145.79.227.31:4797 , piece_id=KRMTNOTWOWCIERSEUWBWZOIPDDUWYZKUBX74ZAIWV737CIKFVRLA, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.181351754 (213h53m40.052434789s) [noise=true , address=145.79.227.51:4411 , piece_id=V7ISMKJUJ6EDCLYOJIZBCBUUNDIWADZPZUUY7XH2F6BHVZILRXBQ, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.181709967 (213h53m40.052793002s) [noise=true , address=145.79.227.46:4601 , piece_id=AFT773CWP62ANAIFPUTDYQ35GSF6K2NFEAO7X4VKIT62CQD3NJEQ, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.183204752 (213h53m40.054287787s) [noise=true , address=145.79.227.28:4725 , piece_id=GNFCQWHHQHCCHL7VNU6TYNEFJZFGR7SMXQO66UCSNORVK4XAHPGA, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.183813983 (213h53m40.054897028s) [noise=true , address=145.79.227.45:4229 , piece_id=4IHXIUBBXYN6MZYQBRBRMJZ5HYOFNCLHKOVZKFVS6M37QMRE35OA, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.186380270 (213h53m40.057463315s) [noise=true , address=145.79.227.38:4978 , piece_id=3AV4CLNBUBIXI73A3YSDKOPIMOROFBI73TVA3KNGDSBSE6KMYNEA, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.198000036 (213h53m40.069083071s) [noise=true , address=145.79.227.36:4519 , piece_id=5X7PDP7FELCD6A5566SD7YBEUN4WELNB4IDV545B27TJ6DPEIIKA, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.201922084 (213h53m40.07300513s) [noise=true , address=145.79.227.47:4267 , piece_id=FOKTCGQ7VI4GDXRJ64LDLVSALAHAL7GRFI257S6X6G2Y4IGUNW6Q, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.217629133 (213h53m40.088712178s) [noise=true , address=145.79.227.34:4905 , piece_id=GHVXBDDAG64LE2A5RRGK3POPOL3QSQGGI7UH3NIHEZTEG7A7GODA, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.217662106 (213h53m40.088745151s) [seg_pos={0 0}, upload=187397]: Segment reached optimal threshold of 13 pieces. Dec 14 13:22:40.218130388 (213h53m40.089213423s) [noise=true , address=145.79.227.44:4193 , piece_id=UMQOWDMKE3PCVS6DQUPMMRR3IYW7EUY3ABOGSFVJ3H7KGFRUZ7HA, seg_pos={0 0}, upload=187397]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8): protocol: expected piece hash; context canceled; EOF Dec 14 13:22:40.218148793 (213h53m40.089231828s) [noise=true , address=145.79.227.44:4193 , piece_id=UMQOWDMKE3PCVS6DQUPMMRR3IYW7EUY3ABOGSFVJ3H7KGFRUZ7HA, seg_pos={0 0}, upload=187397]: Long tail cancellation due to optimal threshold Dec 14 13:22:40.218265846 (213h53m40.089348891s) [noise=true , address=145.79.227.35:4481 , piece_id=LTU57UBRXVWPXJKKRNWYSIQHL7YYPNDMI7R5GARIP55AO7KS4TXQ, seg_pos={0 0}, upload=187397]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bRK8WWmAnWUwzCfHu8JtZgrruYSJCvhnsYTKcaLccJcv5RQjp): protocol: expected piece hash; context canceled; EOF Dec 14 13:22:40.218275184 (213h53m40.089358239s) [noise=true , address=145.79.227.35:4481 , piece_id=LTU57UBRXVWPXJKKRNWYSIQHL7YYPNDMI7R5GARIP55AO7KS4TXQ, seg_pos={0 0}, upload=187397]: Long tail cancellation due to optimal threshold Dec 14 13:22:40.218315581 (213h53m40.089398616s) [noise=true , address=145.79.227.42:4123 , piece_id=ES2R3XH3RQ2UAJTCWRLZBY3VHUZQVUA2EMCNBYBQSTR3N7NWU6EQ, seg_pos={0 0}, upload=187397]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM): protocol: expected piece hash; context canceled; EOF Dec 14 13:22:40.218330358 (213h53m40.089413393s) [noise=true , address=145.79.227.42:4123 , piece_id=ES2R3XH3RQ2UAJTCWRLZBY3VHUZQVUA2EMCNBYBQSTR3N7NWU6EQ, seg_pos={0 0}, upload=187397]: Long tail cancellation due to optimal threshold Dec 14 13:22:40.218347892 (213h53m40.089430937s) [seg_pos={0 0}, upload=187397]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 14 13:22:40.218452501 (213h53m40.089535556s) [upload=187397]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 14 13:22:40.218479914 (213h53m40.089562969s) [upload=187397]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 14 13:22:43.384827291 (213h53m43.255910336s) [upload=187397]: Done uploading segments... Dec 14 13:22:43.384857198 (213h53m43.255940233s) [upload=187397]: Upload finished. err: <nil> Duration: 8.850897777s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [........X...................................X...............] duration: 24.07715434s trace: https://jaeger.dev.storj.io/trace/2f31d4ad7317aa95 FAIL (2025-12-14 13:39:04 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 24.077188414s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [............................................X.......X...X..X] duration: 5.001710362s trace: https://jaeger.dev.storj.io/trace/3891d01bf25a7e1e FAIL (2025-12-14 13:38:43 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001738396s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 23.617069144s trace: https://jaeger.dev.storj.io/trace/8ca15e950587d46 FAIL (2025-12-14 13:38:41 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12YKLTah3swX14kL82eNogJxd8D8Hjfq7SPjDDQAapDnXmDctP1, Piece ID: GNFCQWHHQHCCHL7VNU6TYNEFJZFGR7SMXQO66UCSNORVK4XAHPGA) context deadline exceeded; piecestore: (Node ID: 125GxLsL5UtXkuJ1LFvF42N9iZkKtRK3rq63xdR25JQbtwoT1L5, Piece ID: AFT773CWP62ANAIFPUTDYQ35GSF6K2NFEAO7X4VKIT62CQD3NJEQ) context deadline exceeded ======LOGS====== Dec 14 13:22:35.112522281 (213h53m34.983605316s) [upload=187397]: Starting upload Dec 14 13:22:35.112587294 (213h53m34.983670329s) [upload=187397]: Done starting upload Dec 14 13:22:35.112673899 (213h53m34.983756934s) [upload=187397]: Uploading segments... Dec 14 13:22:35.112767347 (213h53m34.983850393s) [upload=187397]: Got next segment. Inline: false Dec 14 13:22:35.112800511 (213h53m34.983883556s) [upload=187397]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 14 13:22:35.112805821 (213h53m34.983888856s) [upload=187397]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 14 13:22:37.194903810 (213h53m37.065986845s) [seg_pos={0 0}, upload=187397]: Begin upload segment... Dec 14 13:22:37.195008499 (213h53m37.066091534s) [noise=true , address=145.79.227.42:4123 , piece_id=ES2R3XH3RQ2UAJTCWRLZBY3VHUZQVUA2EMCNBYBQSTR3N7NWU6EQ, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.195025751 (213h53m37.066108786s) [seg_pos={0 0}, upload=187397]: Done begin upload segment. Dec 14 13:22:37.195062162 (213h53m37.066145197s) [noise=true , address=145.79.227.44:4193 , piece_id=UMQOWDMKE3PCVS6DQUPMMRR3IYW7EUY3ABOGSFVJ3H7KGFRUZ7HA, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.195125401 (213h53m37.066208446s) [noise=true , address=145.79.227.45:4229 , piece_id=4IHXIUBBXYN6MZYQBRBRMJZ5HYOFNCLHKOVZKFVS6M37QMRE35OA, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.195764098 (213h53m37.066847133s) [noise=true , address=145.79.227.35:4482 , piece_id=RAUYG5T6DQIDOV5QM2OCE4GAPYIFBHUA3ZZR3Y6KN7ZX4OPMPEZA, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.196145995 (213h53m37.06722904s) [noise=true , address=145.79.227.47:4267 , piece_id=FOKTCGQ7VI4GDXRJ64LDLVSALAHAL7GRFI257S6X6G2Y4IGUNW6Q, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.196522081 (213h53m37.067605116s) [noise=true , address=145.79.227.36:4519 , piece_id=5X7PDP7FELCD6A5566SD7YBEUN4WELNB4IDV545B27TJ6DPEIIKA, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.196919738 (213h53m37.068002783s) [noise=true , address=145.79.227.40:4050 , piece_id=OFTY43B5JLDTXZRWXEHQZLUP6YJOL75CAL36EHSOQJR3GIHZOAJQ, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.197734449 (213h53m37.068817484s) [noise=true , address=145.79.227.28:4725 , piece_id=GNFCQWHHQHCCHL7VNU6TYNEFJZFGR7SMXQO66UCSNORVK4XAHPGA, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.197786599 (213h53m37.068869634s) [noise=true , address=145.79.227.43:4158 , piece_id=HDWGVVZAZI3R6KQFIUG74SWRF3G2USEMNKVNMKZDYISPRA74AUGQ, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.198256694 (213h53m37.069339739s) [noise=true , address=145.79.227.35:4481 , piece_id=LTU57UBRXVWPXJKKRNWYSIQHL7YYPNDMI7R5GARIP55AO7KS4TXQ, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.198675992 (213h53m37.069759027s) [noise=true , address=145.79.227.51:4411 , piece_id=V7ISMKJUJ6EDCLYOJIZBCBUUNDIWADZPZUUY7XH2F6BHVZILRXBQ, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.199111532 (213h53m37.070194567s) [noise=true , address=145.79.227.26:4655 , piece_id=NSYNQURKGVD7VTPMXJGF5RDIQKNUBLVKFYUE5CLPQ6IMIEJZQ4SQ, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.199658874 (213h53m37.07074191s) [noise=true , address=145.79.227.34:4905 , piece_id=GHVXBDDAG64LE2A5RRGK3POPOL3QSQGGI7UH3NIHEZTEG7A7GODA, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.200449229 (213h53m37.071532274s) [noise=true , address=145.79.227.38:4978 , piece_id=3AV4CLNBUBIXI73A3YSDKOPIMOROFBI73TVA3KNGDSBSE6KMYNEA, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.200877725 (213h53m37.07196077s) [noise=true , address=145.79.227.46:4601 , piece_id=AFT773CWP62ANAIFPUTDYQ35GSF6K2NFEAO7X4VKIT62CQD3NJEQ, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.207413279 (213h53m37.078496314s) [noise=true , address=145.79.227.31:4797 , piece_id=KRMTNOTWOWCIERSEUWBWZOIPDDUWYZKUBX74ZAIWV737CIKFVRLA, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:39.731884038 (213h53m39.602967073s) [upload=187397]: Next returned nil segment Dec 14 13:22:39.731901430 (213h53m39.602984475s) [upload=187397]: Waiting for error group managing segments... Dec 14 13:22:40.154962424 (213h53m40.026045469s) [noise=true , address=145.79.227.26:4655 , piece_id=NSYNQURKGVD7VTPMXJGF5RDIQKNUBLVKFYUE5CLPQ6IMIEJZQ4SQ, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.159418800 (213h53m40.030501835s) [noise=true , address=145.79.227.40:4050 , piece_id=OFTY43B5JLDTXZRWXEHQZLUP6YJOL75CAL36EHSOQJR3GIHZOAJQ, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.175663783 (213h53m40.046746829s) [noise=true , address=145.79.227.43:4158 , piece_id=HDWGVVZAZI3R6KQFIUG74SWRF3G2USEMNKVNMKZDYISPRA74AUGQ, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.175938396 (213h53m40.047021432s) [noise=true , address=145.79.227.35:4482 , piece_id=RAUYG5T6DQIDOV5QM2OCE4GAPYIFBHUA3ZZR3Y6KN7ZX4OPMPEZA, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.177599980 (213h53m40.048683025s) [noise=true , address=145.79.227.31:4797 , piece_id=KRMTNOTWOWCIERSEUWBWZOIPDDUWYZKUBX74ZAIWV737CIKFVRLA, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.181351754 (213h53m40.052434789s) [noise=true , address=145.79.227.51:4411 , piece_id=V7ISMKJUJ6EDCLYOJIZBCBUUNDIWADZPZUUY7XH2F6BHVZILRXBQ, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.181709967 (213h53m40.052793002s) [noise=true , address=145.79.227.46:4601 , piece_id=AFT773CWP62ANAIFPUTDYQ35GSF6K2NFEAO7X4VKIT62CQD3NJEQ, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.183204752 (213h53m40.054287787s) [noise=true , address=145.79.227.28:4725 , piece_id=GNFCQWHHQHCCHL7VNU6TYNEFJZFGR7SMXQO66UCSNORVK4XAHPGA, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.183813983 (213h53m40.054897028s) [noise=true , address=145.79.227.45:4229 , piece_id=4IHXIUBBXYN6MZYQBRBRMJZ5HYOFNCLHKOVZKFVS6M37QMRE35OA, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.186380270 (213h53m40.057463315s) [noise=true , address=145.79.227.38:4978 , piece_id=3AV4CLNBUBIXI73A3YSDKOPIMOROFBI73TVA3KNGDSBSE6KMYNEA, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.198000036 (213h53m40.069083071s) [noise=true , address=145.79.227.36:4519 , piece_id=5X7PDP7FELCD6A5566SD7YBEUN4WELNB4IDV545B27TJ6DPEIIKA, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.201922084 (213h53m40.07300513s) [noise=true , address=145.79.227.47:4267 , piece_id=FOKTCGQ7VI4GDXRJ64LDLVSALAHAL7GRFI257S6X6G2Y4IGUNW6Q, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.217629133 (213h53m40.088712178s) [noise=true , address=145.79.227.34:4905 , piece_id=GHVXBDDAG64LE2A5RRGK3POPOL3QSQGGI7UH3NIHEZTEG7A7GODA, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.217662106 (213h53m40.088745151s) [seg_pos={0 0}, upload=187397]: Segment reached optimal threshold of 13 pieces. Dec 14 13:22:40.218130388 (213h53m40.089213423s) [noise=true , address=145.79.227.44:4193 , piece_id=UMQOWDMKE3PCVS6DQUPMMRR3IYW7EUY3ABOGSFVJ3H7KGFRUZ7HA, seg_pos={0 0}, upload=187397]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8): protocol: expected piece hash; context canceled; EOF Dec 14 13:22:40.218148793 (213h53m40.089231828s) [noise=true , address=145.79.227.44:4193 , piece_id=UMQOWDMKE3PCVS6DQUPMMRR3IYW7EUY3ABOGSFVJ3H7KGFRUZ7HA, seg_pos={0 0}, upload=187397]: Long tail cancellation due to optimal threshold Dec 14 13:22:40.218265846 (213h53m40.089348891s) [noise=true , address=145.79.227.35:4481 , piece_id=LTU57UBRXVWPXJKKRNWYSIQHL7YYPNDMI7R5GARIP55AO7KS4TXQ, seg_pos={0 0}, upload=187397]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bRK8WWmAnWUwzCfHu8JtZgrruYSJCvhnsYTKcaLccJcv5RQjp): protocol: expected piece hash; context canceled; EOF Dec 14 13:22:40.218275184 (213h53m40.089358239s) [noise=true , address=145.79.227.35:4481 , piece_id=LTU57UBRXVWPXJKKRNWYSIQHL7YYPNDMI7R5GARIP55AO7KS4TXQ, seg_pos={0 0}, upload=187397]: Long tail cancellation due to optimal threshold Dec 14 13:22:40.218315581 (213h53m40.089398616s) [noise=true , address=145.79.227.42:4123 , piece_id=ES2R3XH3RQ2UAJTCWRLZBY3VHUZQVUA2EMCNBYBQSTR3N7NWU6EQ, seg_pos={0 0}, upload=187397]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM): protocol: expected piece hash; context canceled; EOF Dec 14 13:22:40.218330358 (213h53m40.089413393s) [noise=true , address=145.79.227.42:4123 , piece_id=ES2R3XH3RQ2UAJTCWRLZBY3VHUZQVUA2EMCNBYBQSTR3N7NWU6EQ, seg_pos={0 0}, upload=187397]: Long tail cancellation due to optimal threshold Dec 14 13:22:40.218347892 (213h53m40.089430937s) [seg_pos={0 0}, upload=187397]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 14 13:22:40.218452501 (213h53m40.089535556s) [upload=187397]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 14 13:22:40.218479914 (213h53m40.089562969s) [upload=187397]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 14 13:22:43.384827291 (213h53m43.255910336s) [upload=187397]: Done uploading segments... Dec 14 13:22:43.384857198 (213h53m43.255940233s) [upload=187397]: Upload finished. err: <nil> Duration: 23.617094992s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.........X...................................X..............] duration: 22.789845941s trace: https://jaeger.dev.storj.io/trace/2f31d4ad7317aa95 FAIL (2025-12-14 13:38:23 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 22.789896228s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 10.138393866s trace: https://jaeger.dev.storj.io/trace/8ca15e950587d46 FAIL (2025-12-14 13:38:08 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12YKLTah3swX14kL82eNogJxd8D8Hjfq7SPjDDQAapDnXmDctP1, Piece ID: GNFCQWHHQHCCHL7VNU6TYNEFJZFGR7SMXQO66UCSNORVK4XAHPGA) context deadline exceeded; piecestore: (Node ID: 125GxLsL5UtXkuJ1LFvF42N9iZkKtRK3rq63xdR25JQbtwoT1L5, Piece ID: AFT773CWP62ANAIFPUTDYQ35GSF6K2NFEAO7X4VKIT62CQD3NJEQ) context deadline exceeded ======LOGS====== Dec 14 13:22:35.112522281 (213h53m34.983605316s) [upload=187397]: Starting upload Dec 14 13:22:35.112587294 (213h53m34.983670329s) [upload=187397]: Done starting upload Dec 14 13:22:35.112673899 (213h53m34.983756934s) [upload=187397]: Uploading segments... Dec 14 13:22:35.112767347 (213h53m34.983850393s) [upload=187397]: Got next segment. Inline: false Dec 14 13:22:35.112800511 (213h53m34.983883556s) [upload=187397]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 14 13:22:35.112805821 (213h53m34.983888856s) [upload=187397]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 14 13:22:37.194903810 (213h53m37.065986845s) [seg_pos={0 0}, upload=187397]: Begin upload segment... Dec 14 13:22:37.195008499 (213h53m37.066091534s) [noise=true , address=145.79.227.42:4123 , piece_id=ES2R3XH3RQ2UAJTCWRLZBY3VHUZQVUA2EMCNBYBQSTR3N7NWU6EQ, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.195025751 (213h53m37.066108786s) [seg_pos={0 0}, upload=187397]: Done begin upload segment. Dec 14 13:22:37.195062162 (213h53m37.066145197s) [noise=true , address=145.79.227.44:4193 , piece_id=UMQOWDMKE3PCVS6DQUPMMRR3IYW7EUY3ABOGSFVJ3H7KGFRUZ7HA, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.195125401 (213h53m37.066208446s) [noise=true , address=145.79.227.45:4229 , piece_id=4IHXIUBBXYN6MZYQBRBRMJZ5HYOFNCLHKOVZKFVS6M37QMRE35OA, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.195764098 (213h53m37.066847133s) [noise=true , address=145.79.227.35:4482 , piece_id=RAUYG5T6DQIDOV5QM2OCE4GAPYIFBHUA3ZZR3Y6KN7ZX4OPMPEZA, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.196145995 (213h53m37.06722904s) [noise=true , address=145.79.227.47:4267 , piece_id=FOKTCGQ7VI4GDXRJ64LDLVSALAHAL7GRFI257S6X6G2Y4IGUNW6Q, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.196522081 (213h53m37.067605116s) [noise=true , address=145.79.227.36:4519 , piece_id=5X7PDP7FELCD6A5566SD7YBEUN4WELNB4IDV545B27TJ6DPEIIKA, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.196919738 (213h53m37.068002783s) [noise=true , address=145.79.227.40:4050 , piece_id=OFTY43B5JLDTXZRWXEHQZLUP6YJOL75CAL36EHSOQJR3GIHZOAJQ, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.197734449 (213h53m37.068817484s) [noise=true , address=145.79.227.28:4725 , piece_id=GNFCQWHHQHCCHL7VNU6TYNEFJZFGR7SMXQO66UCSNORVK4XAHPGA, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.197786599 (213h53m37.068869634s) [noise=true , address=145.79.227.43:4158 , piece_id=HDWGVVZAZI3R6KQFIUG74SWRF3G2USEMNKVNMKZDYISPRA74AUGQ, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.198256694 (213h53m37.069339739s) [noise=true , address=145.79.227.35:4481 , piece_id=LTU57UBRXVWPXJKKRNWYSIQHL7YYPNDMI7R5GARIP55AO7KS4TXQ, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.198675992 (213h53m37.069759027s) [noise=true , address=145.79.227.51:4411 , piece_id=V7ISMKJUJ6EDCLYOJIZBCBUUNDIWADZPZUUY7XH2F6BHVZILRXBQ, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.199111532 (213h53m37.070194567s) [noise=true , address=145.79.227.26:4655 , piece_id=NSYNQURKGVD7VTPMXJGF5RDIQKNUBLVKFYUE5CLPQ6IMIEJZQ4SQ, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.199658874 (213h53m37.07074191s) [noise=true , address=145.79.227.34:4905 , piece_id=GHVXBDDAG64LE2A5RRGK3POPOL3QSQGGI7UH3NIHEZTEG7A7GODA, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.200449229 (213h53m37.071532274s) [noise=true , address=145.79.227.38:4978 , piece_id=3AV4CLNBUBIXI73A3YSDKOPIMOROFBI73TVA3KNGDSBSE6KMYNEA, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.200877725 (213h53m37.07196077s) [noise=true , address=145.79.227.46:4601 , piece_id=AFT773CWP62ANAIFPUTDYQ35GSF6K2NFEAO7X4VKIT62CQD3NJEQ, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.207413279 (213h53m37.078496314s) [noise=true , address=145.79.227.31:4797 , piece_id=KRMTNOTWOWCIERSEUWBWZOIPDDUWYZKUBX74ZAIWV737CIKFVRLA, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:39.731884038 (213h53m39.602967073s) [upload=187397]: Next returned nil segment Dec 14 13:22:39.731901430 (213h53m39.602984475s) [upload=187397]: Waiting for error group managing segments... Dec 14 13:22:40.154962424 (213h53m40.026045469s) [noise=true , address=145.79.227.26:4655 , piece_id=NSYNQURKGVD7VTPMXJGF5RDIQKNUBLVKFYUE5CLPQ6IMIEJZQ4SQ, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.159418800 (213h53m40.030501835s) [noise=true , address=145.79.227.40:4050 , piece_id=OFTY43B5JLDTXZRWXEHQZLUP6YJOL75CAL36EHSOQJR3GIHZOAJQ, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.175663783 (213h53m40.046746829s) [noise=true , address=145.79.227.43:4158 , piece_id=HDWGVVZAZI3R6KQFIUG74SWRF3G2USEMNKVNMKZDYISPRA74AUGQ, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.175938396 (213h53m40.047021432s) [noise=true , address=145.79.227.35:4482 , piece_id=RAUYG5T6DQIDOV5QM2OCE4GAPYIFBHUA3ZZR3Y6KN7ZX4OPMPEZA, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.177599980 (213h53m40.048683025s) [noise=true , address=145.79.227.31:4797 , piece_id=KRMTNOTWOWCIERSEUWBWZOIPDDUWYZKUBX74ZAIWV737CIKFVRLA, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.181351754 (213h53m40.052434789s) [noise=true , address=145.79.227.51:4411 , piece_id=V7ISMKJUJ6EDCLYOJIZBCBUUNDIWADZPZUUY7XH2F6BHVZILRXBQ, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.181709967 (213h53m40.052793002s) [noise=true , address=145.79.227.46:4601 , piece_id=AFT773CWP62ANAIFPUTDYQ35GSF6K2NFEAO7X4VKIT62CQD3NJEQ, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.183204752 (213h53m40.054287787s) [noise=true , address=145.79.227.28:4725 , piece_id=GNFCQWHHQHCCHL7VNU6TYNEFJZFGR7SMXQO66UCSNORVK4XAHPGA, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.183813983 (213h53m40.054897028s) [noise=true , address=145.79.227.45:4229 , piece_id=4IHXIUBBXYN6MZYQBRBRMJZ5HYOFNCLHKOVZKFVS6M37QMRE35OA, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.186380270 (213h53m40.057463315s) [noise=true , address=145.79.227.38:4978 , piece_id=3AV4CLNBUBIXI73A3YSDKOPIMOROFBI73TVA3KNGDSBSE6KMYNEA, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.198000036 (213h53m40.069083071s) [noise=true , address=145.79.227.36:4519 , piece_id=5X7PDP7FELCD6A5566SD7YBEUN4WELNB4IDV545B27TJ6DPEIIKA, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.201922084 (213h53m40.07300513s) [noise=true , address=145.79.227.47:4267 , piece_id=FOKTCGQ7VI4GDXRJ64LDLVSALAHAL7GRFI257S6X6G2Y4IGUNW6Q, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.217629133 (213h53m40.088712178s) [noise=true , address=145.79.227.34:4905 , piece_id=GHVXBDDAG64LE2A5RRGK3POPOL3QSQGGI7UH3NIHEZTEG7A7GODA, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.217662106 (213h53m40.088745151s) [seg_pos={0 0}, upload=187397]: Segment reached optimal threshold of 13 pieces. Dec 14 13:22:40.218130388 (213h53m40.089213423s) [noise=true , address=145.79.227.44:4193 , piece_id=UMQOWDMKE3PCVS6DQUPMMRR3IYW7EUY3ABOGSFVJ3H7KGFRUZ7HA, seg_pos={0 0}, upload=187397]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8): protocol: expected piece hash; context canceled; EOF Dec 14 13:22:40.218148793 (213h53m40.089231828s) [noise=true , address=145.79.227.44:4193 , piece_id=UMQOWDMKE3PCVS6DQUPMMRR3IYW7EUY3ABOGSFVJ3H7KGFRUZ7HA, seg_pos={0 0}, upload=187397]: Long tail cancellation due to optimal threshold Dec 14 13:22:40.218265846 (213h53m40.089348891s) [noise=true , address=145.79.227.35:4481 , piece_id=LTU57UBRXVWPXJKKRNWYSIQHL7YYPNDMI7R5GARIP55AO7KS4TXQ, seg_pos={0 0}, upload=187397]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bRK8WWmAnWUwzCfHu8JtZgrruYSJCvhnsYTKcaLccJcv5RQjp): protocol: expected piece hash; context canceled; EOF Dec 14 13:22:40.218275184 (213h53m40.089358239s) [noise=true , address=145.79.227.35:4481 , piece_id=LTU57UBRXVWPXJKKRNWYSIQHL7YYPNDMI7R5GARIP55AO7KS4TXQ, seg_pos={0 0}, upload=187397]: Long tail cancellation due to optimal threshold Dec 14 13:22:40.218315581 (213h53m40.089398616s) [noise=true , address=145.79.227.42:4123 , piece_id=ES2R3XH3RQ2UAJTCWRLZBY3VHUZQVUA2EMCNBYBQSTR3N7NWU6EQ, seg_pos={0 0}, upload=187397]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM): protocol: expected piece hash; context canceled; EOF Dec 14 13:22:40.218330358 (213h53m40.089413393s) [noise=true , address=145.79.227.42:4123 , piece_id=ES2R3XH3RQ2UAJTCWRLZBY3VHUZQVUA2EMCNBYBQSTR3N7NWU6EQ, seg_pos={0 0}, upload=187397]: Long tail cancellation due to optimal threshold Dec 14 13:22:40.218347892 (213h53m40.089430937s) [seg_pos={0 0}, upload=187397]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 14 13:22:40.218452501 (213h53m40.089535556s) [upload=187397]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 14 13:22:40.218479914 (213h53m40.089562969s) [upload=187397]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 14 13:22:43.384827291 (213h53m43.255910336s) [upload=187397]: Done uploading segments... Dec 14 13:22:43.384857198 (213h53m43.255940233s) [upload=187397]: Upload finished. err: <nil> Duration: 10.138426378s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [.............................................X.......X...X..] duration: 4.814565968s trace: https://jaeger.dev.storj.io/trace/20dc4c4ab3a6940f FAIL (2025-12-14 13:37:53 UTC) orbiter warning: context deadline exceeded Duration: 4.814590914s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 13.487491477s trace: https://jaeger.dev.storj.io/trace/8ca15e950587d46 FAIL (2025-12-14 13:37:45 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12YKLTah3swX14kL82eNogJxd8D8Hjfq7SPjDDQAapDnXmDctP1, Piece ID: GNFCQWHHQHCCHL7VNU6TYNEFJZFGR7SMXQO66UCSNORVK4XAHPGA) context deadline exceeded; piecestore: (Node ID: 125GxLsL5UtXkuJ1LFvF42N9iZkKtRK3rq63xdR25JQbtwoT1L5, Piece ID: AFT773CWP62ANAIFPUTDYQ35GSF6K2NFEAO7X4VKIT62CQD3NJEQ) context deadline exceeded ======LOGS====== Dec 14 13:22:35.112522281 (213h53m34.983605316s) [upload=187397]: Starting upload Dec 14 13:22:35.112587294 (213h53m34.983670329s) [upload=187397]: Done starting upload Dec 14 13:22:35.112673899 (213h53m34.983756934s) [upload=187397]: Uploading segments... Dec 14 13:22:35.112767347 (213h53m34.983850393s) [upload=187397]: Got next segment. Inline: false Dec 14 13:22:35.112800511 (213h53m34.983883556s) [upload=187397]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 14 13:22:35.112805821 (213h53m34.983888856s) [upload=187397]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 14 13:22:37.194903810 (213h53m37.065986845s) [seg_pos={0 0}, upload=187397]: Begin upload segment... Dec 14 13:22:37.195008499 (213h53m37.066091534s) [noise=true , address=145.79.227.42:4123 , piece_id=ES2R3XH3RQ2UAJTCWRLZBY3VHUZQVUA2EMCNBYBQSTR3N7NWU6EQ, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.195025751 (213h53m37.066108786s) [seg_pos={0 0}, upload=187397]: Done begin upload segment. Dec 14 13:22:37.195062162 (213h53m37.066145197s) [noise=true , address=145.79.227.44:4193 , piece_id=UMQOWDMKE3PCVS6DQUPMMRR3IYW7EUY3ABOGSFVJ3H7KGFRUZ7HA, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.195125401 (213h53m37.066208446s) [noise=true , address=145.79.227.45:4229 , piece_id=4IHXIUBBXYN6MZYQBRBRMJZ5HYOFNCLHKOVZKFVS6M37QMRE35OA, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.195764098 (213h53m37.066847133s) [noise=true , address=145.79.227.35:4482 , piece_id=RAUYG5T6DQIDOV5QM2OCE4GAPYIFBHUA3ZZR3Y6KN7ZX4OPMPEZA, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.196145995 (213h53m37.06722904s) [noise=true , address=145.79.227.47:4267 , piece_id=FOKTCGQ7VI4GDXRJ64LDLVSALAHAL7GRFI257S6X6G2Y4IGUNW6Q, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.196522081 (213h53m37.067605116s) [noise=true , address=145.79.227.36:4519 , piece_id=5X7PDP7FELCD6A5566SD7YBEUN4WELNB4IDV545B27TJ6DPEIIKA, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.196919738 (213h53m37.068002783s) [noise=true , address=145.79.227.40:4050 , piece_id=OFTY43B5JLDTXZRWXEHQZLUP6YJOL75CAL36EHSOQJR3GIHZOAJQ, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.197734449 (213h53m37.068817484s) [noise=true , address=145.79.227.28:4725 , piece_id=GNFCQWHHQHCCHL7VNU6TYNEFJZFGR7SMXQO66UCSNORVK4XAHPGA, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.197786599 (213h53m37.068869634s) [noise=true , address=145.79.227.43:4158 , piece_id=HDWGVVZAZI3R6KQFIUG74SWRF3G2USEMNKVNMKZDYISPRA74AUGQ, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.198256694 (213h53m37.069339739s) [noise=true , address=145.79.227.35:4481 , piece_id=LTU57UBRXVWPXJKKRNWYSIQHL7YYPNDMI7R5GARIP55AO7KS4TXQ, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.198675992 (213h53m37.069759027s) [noise=true , address=145.79.227.51:4411 , piece_id=V7ISMKJUJ6EDCLYOJIZBCBUUNDIWADZPZUUY7XH2F6BHVZILRXBQ, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.199111532 (213h53m37.070194567s) [noise=true , address=145.79.227.26:4655 , piece_id=NSYNQURKGVD7VTPMXJGF5RDIQKNUBLVKFYUE5CLPQ6IMIEJZQ4SQ, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.199658874 (213h53m37.07074191s) [noise=true , address=145.79.227.34:4905 , piece_id=GHVXBDDAG64LE2A5RRGK3POPOL3QSQGGI7UH3NIHEZTEG7A7GODA, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.200449229 (213h53m37.071532274s) [noise=true , address=145.79.227.38:4978 , piece_id=3AV4CLNBUBIXI73A3YSDKOPIMOROFBI73TVA3KNGDSBSE6KMYNEA, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.200877725 (213h53m37.07196077s) [noise=true , address=145.79.227.46:4601 , piece_id=AFT773CWP62ANAIFPUTDYQ35GSF6K2NFEAO7X4VKIT62CQD3NJEQ, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.207413279 (213h53m37.078496314s) [noise=true , address=145.79.227.31:4797 , piece_id=KRMTNOTWOWCIERSEUWBWZOIPDDUWYZKUBX74ZAIWV737CIKFVRLA, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:39.731884038 (213h53m39.602967073s) [upload=187397]: Next returned nil segment Dec 14 13:22:39.731901430 (213h53m39.602984475s) [upload=187397]: Waiting for error group managing segments... Dec 14 13:22:40.154962424 (213h53m40.026045469s) [noise=true , address=145.79.227.26:4655 , piece_id=NSYNQURKGVD7VTPMXJGF5RDIQKNUBLVKFYUE5CLPQ6IMIEJZQ4SQ, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.159418800 (213h53m40.030501835s) [noise=true , address=145.79.227.40:4050 , piece_id=OFTY43B5JLDTXZRWXEHQZLUP6YJOL75CAL36EHSOQJR3GIHZOAJQ, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.175663783 (213h53m40.046746829s) [noise=true , address=145.79.227.43:4158 , piece_id=HDWGVVZAZI3R6KQFIUG74SWRF3G2USEMNKVNMKZDYISPRA74AUGQ, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.175938396 (213h53m40.047021432s) [noise=true , address=145.79.227.35:4482 , piece_id=RAUYG5T6DQIDOV5QM2OCE4GAPYIFBHUA3ZZR3Y6KN7ZX4OPMPEZA, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.177599980 (213h53m40.048683025s) [noise=true , address=145.79.227.31:4797 , piece_id=KRMTNOTWOWCIERSEUWBWZOIPDDUWYZKUBX74ZAIWV737CIKFVRLA, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.181351754 (213h53m40.052434789s) [noise=true , address=145.79.227.51:4411 , piece_id=V7ISMKJUJ6EDCLYOJIZBCBUUNDIWADZPZUUY7XH2F6BHVZILRXBQ, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.181709967 (213h53m40.052793002s) [noise=true , address=145.79.227.46:4601 , piece_id=AFT773CWP62ANAIFPUTDYQ35GSF6K2NFEAO7X4VKIT62CQD3NJEQ, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.183204752 (213h53m40.054287787s) [noise=true , address=145.79.227.28:4725 , piece_id=GNFCQWHHQHCCHL7VNU6TYNEFJZFGR7SMXQO66UCSNORVK4XAHPGA, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.183813983 (213h53m40.054897028s) [noise=true , address=145.79.227.45:4229 , piece_id=4IHXIUBBXYN6MZYQBRBRMJZ5HYOFNCLHKOVZKFVS6M37QMRE35OA, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.186380270 (213h53m40.057463315s) [noise=true , address=145.79.227.38:4978 , piece_id=3AV4CLNBUBIXI73A3YSDKOPIMOROFBI73TVA3KNGDSBSE6KMYNEA, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.198000036 (213h53m40.069083071s) [noise=true , address=145.79.227.36:4519 , piece_id=5X7PDP7FELCD6A5566SD7YBEUN4WELNB4IDV545B27TJ6DPEIIKA, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.201922084 (213h53m40.07300513s) [noise=true , address=145.79.227.47:4267 , piece_id=FOKTCGQ7VI4GDXRJ64LDLVSALAHAL7GRFI257S6X6G2Y4IGUNW6Q, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.217629133 (213h53m40.088712178s) [noise=true , address=145.79.227.34:4905 , piece_id=GHVXBDDAG64LE2A5RRGK3POPOL3QSQGGI7UH3NIHEZTEG7A7GODA, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.217662106 (213h53m40.088745151s) [seg_pos={0 0}, upload=187397]: Segment reached optimal threshold of 13 pieces. Dec 14 13:22:40.218130388 (213h53m40.089213423s) [noise=true , address=145.79.227.44:4193 , piece_id=UMQOWDMKE3PCVS6DQUPMMRR3IYW7EUY3ABOGSFVJ3H7KGFRUZ7HA, seg_pos={0 0}, upload=187397]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8): protocol: expected piece hash; context canceled; EOF Dec 14 13:22:40.218148793 (213h53m40.089231828s) [noise=true , address=145.79.227.44:4193 , piece_id=UMQOWDMKE3PCVS6DQUPMMRR3IYW7EUY3ABOGSFVJ3H7KGFRUZ7HA, seg_pos={0 0}, upload=187397]: Long tail cancellation due to optimal threshold Dec 14 13:22:40.218265846 (213h53m40.089348891s) [noise=true , address=145.79.227.35:4481 , piece_id=LTU57UBRXVWPXJKKRNWYSIQHL7YYPNDMI7R5GARIP55AO7KS4TXQ, seg_pos={0 0}, upload=187397]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bRK8WWmAnWUwzCfHu8JtZgrruYSJCvhnsYTKcaLccJcv5RQjp): protocol: expected piece hash; context canceled; EOF Dec 14 13:22:40.218275184 (213h53m40.089358239s) [noise=true , address=145.79.227.35:4481 , piece_id=LTU57UBRXVWPXJKKRNWYSIQHL7YYPNDMI7R5GARIP55AO7KS4TXQ, seg_pos={0 0}, upload=187397]: Long tail cancellation due to optimal threshold Dec 14 13:22:40.218315581 (213h53m40.089398616s) [noise=true , address=145.79.227.42:4123 , piece_id=ES2R3XH3RQ2UAJTCWRLZBY3VHUZQVUA2EMCNBYBQSTR3N7NWU6EQ, seg_pos={0 0}, upload=187397]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM): protocol: expected piece hash; context canceled; EOF Dec 14 13:22:40.218330358 (213h53m40.089413393s) [noise=true , address=145.79.227.42:4123 , piece_id=ES2R3XH3RQ2UAJTCWRLZBY3VHUZQVUA2EMCNBYBQSTR3N7NWU6EQ, seg_pos={0 0}, upload=187397]: Long tail cancellation due to optimal threshold Dec 14 13:22:40.218347892 (213h53m40.089430937s) [seg_pos={0 0}, upload=187397]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 14 13:22:40.218452501 (213h53m40.089535556s) [upload=187397]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 14 13:22:40.218479914 (213h53m40.089562969s) [upload=187397]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 14 13:22:43.384827291 (213h53m43.255910336s) [upload=187397]: Done uploading segments... Dec 14 13:22:43.384857198 (213h53m43.255940233s) [upload=187397]: Upload finished. err: <nil> Duration: 13.487509733s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 20.14811431s trace: https://jaeger.dev.storj.io/trace/8ca15e950587d46 FAIL (2025-12-14 13:37:12 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12YKLTah3swX14kL82eNogJxd8D8Hjfq7SPjDDQAapDnXmDctP1, Piece ID: GNFCQWHHQHCCHL7VNU6TYNEFJZFGR7SMXQO66UCSNORVK4XAHPGA) context deadline exceeded; piecestore: (Node ID: 125GxLsL5UtXkuJ1LFvF42N9iZkKtRK3rq63xdR25JQbtwoT1L5, Piece ID: AFT773CWP62ANAIFPUTDYQ35GSF6K2NFEAO7X4VKIT62CQD3NJEQ) context deadline exceeded ======LOGS====== Dec 14 13:22:35.112522281 (213h53m34.983605316s) [upload=187397]: Starting upload Dec 14 13:22:35.112587294 (213h53m34.983670329s) [upload=187397]: Done starting upload Dec 14 13:22:35.112673899 (213h53m34.983756934s) [upload=187397]: Uploading segments... Dec 14 13:22:35.112767347 (213h53m34.983850393s) [upload=187397]: Got next segment. Inline: false Dec 14 13:22:35.112800511 (213h53m34.983883556s) [upload=187397]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 14 13:22:35.112805821 (213h53m34.983888856s) [upload=187397]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 14 13:22:37.194903810 (213h53m37.065986845s) [seg_pos={0 0}, upload=187397]: Begin upload segment... Dec 14 13:22:37.195008499 (213h53m37.066091534s) [noise=true , address=145.79.227.42:4123 , piece_id=ES2R3XH3RQ2UAJTCWRLZBY3VHUZQVUA2EMCNBYBQSTR3N7NWU6EQ, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.195025751 (213h53m37.066108786s) [seg_pos={0 0}, upload=187397]: Done begin upload segment. Dec 14 13:22:37.195062162 (213h53m37.066145197s) [noise=true , address=145.79.227.44:4193 , piece_id=UMQOWDMKE3PCVS6DQUPMMRR3IYW7EUY3ABOGSFVJ3H7KGFRUZ7HA, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.195125401 (213h53m37.066208446s) [noise=true , address=145.79.227.45:4229 , piece_id=4IHXIUBBXYN6MZYQBRBRMJZ5HYOFNCLHKOVZKFVS6M37QMRE35OA, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.195764098 (213h53m37.066847133s) [noise=true , address=145.79.227.35:4482 , piece_id=RAUYG5T6DQIDOV5QM2OCE4GAPYIFBHUA3ZZR3Y6KN7ZX4OPMPEZA, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.196145995 (213h53m37.06722904s) [noise=true , address=145.79.227.47:4267 , piece_id=FOKTCGQ7VI4GDXRJ64LDLVSALAHAL7GRFI257S6X6G2Y4IGUNW6Q, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.196522081 (213h53m37.067605116s) [noise=true , address=145.79.227.36:4519 , piece_id=5X7PDP7FELCD6A5566SD7YBEUN4WELNB4IDV545B27TJ6DPEIIKA, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.196919738 (213h53m37.068002783s) [noise=true , address=145.79.227.40:4050 , piece_id=OFTY43B5JLDTXZRWXEHQZLUP6YJOL75CAL36EHSOQJR3GIHZOAJQ, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.197734449 (213h53m37.068817484s) [noise=true , address=145.79.227.28:4725 , piece_id=GNFCQWHHQHCCHL7VNU6TYNEFJZFGR7SMXQO66UCSNORVK4XAHPGA, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.197786599 (213h53m37.068869634s) [noise=true , address=145.79.227.43:4158 , piece_id=HDWGVVZAZI3R6KQFIUG74SWRF3G2USEMNKVNMKZDYISPRA74AUGQ, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.198256694 (213h53m37.069339739s) [noise=true , address=145.79.227.35:4481 , piece_id=LTU57UBRXVWPXJKKRNWYSIQHL7YYPNDMI7R5GARIP55AO7KS4TXQ, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.198675992 (213h53m37.069759027s) [noise=true , address=145.79.227.51:4411 , piece_id=V7ISMKJUJ6EDCLYOJIZBCBUUNDIWADZPZUUY7XH2F6BHVZILRXBQ, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.199111532 (213h53m37.070194567s) [noise=true , address=145.79.227.26:4655 , piece_id=NSYNQURKGVD7VTPMXJGF5RDIQKNUBLVKFYUE5CLPQ6IMIEJZQ4SQ, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.199658874 (213h53m37.07074191s) [noise=true , address=145.79.227.34:4905 , piece_id=GHVXBDDAG64LE2A5RRGK3POPOL3QSQGGI7UH3NIHEZTEG7A7GODA, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.200449229 (213h53m37.071532274s) [noise=true , address=145.79.227.38:4978 , piece_id=3AV4CLNBUBIXI73A3YSDKOPIMOROFBI73TVA3KNGDSBSE6KMYNEA, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.200877725 (213h53m37.07196077s) [noise=true , address=145.79.227.46:4601 , piece_id=AFT773CWP62ANAIFPUTDYQ35GSF6K2NFEAO7X4VKIT62CQD3NJEQ, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.207413279 (213h53m37.078496314s) [noise=true , address=145.79.227.31:4797 , piece_id=KRMTNOTWOWCIERSEUWBWZOIPDDUWYZKUBX74ZAIWV737CIKFVRLA, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:39.731884038 (213h53m39.602967073s) [upload=187397]: Next returned nil segment Dec 14 13:22:39.731901430 (213h53m39.602984475s) [upload=187397]: Waiting for error group managing segments... Dec 14 13:22:40.154962424 (213h53m40.026045469s) [noise=true , address=145.79.227.26:4655 , piece_id=NSYNQURKGVD7VTPMXJGF5RDIQKNUBLVKFYUE5CLPQ6IMIEJZQ4SQ, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.159418800 (213h53m40.030501835s) [noise=true , address=145.79.227.40:4050 , piece_id=OFTY43B5JLDTXZRWXEHQZLUP6YJOL75CAL36EHSOQJR3GIHZOAJQ, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.175663783 (213h53m40.046746829s) [noise=true , address=145.79.227.43:4158 , piece_id=HDWGVVZAZI3R6KQFIUG74SWRF3G2USEMNKVNMKZDYISPRA74AUGQ, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.175938396 (213h53m40.047021432s) [noise=true , address=145.79.227.35:4482 , piece_id=RAUYG5T6DQIDOV5QM2OCE4GAPYIFBHUA3ZZR3Y6KN7ZX4OPMPEZA, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.177599980 (213h53m40.048683025s) [noise=true , address=145.79.227.31:4797 , piece_id=KRMTNOTWOWCIERSEUWBWZOIPDDUWYZKUBX74ZAIWV737CIKFVRLA, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.181351754 (213h53m40.052434789s) [noise=true , address=145.79.227.51:4411 , piece_id=V7ISMKJUJ6EDCLYOJIZBCBUUNDIWADZPZUUY7XH2F6BHVZILRXBQ, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.181709967 (213h53m40.052793002s) [noise=true , address=145.79.227.46:4601 , piece_id=AFT773CWP62ANAIFPUTDYQ35GSF6K2NFEAO7X4VKIT62CQD3NJEQ, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.183204752 (213h53m40.054287787s) [noise=true , address=145.79.227.28:4725 , piece_id=GNFCQWHHQHCCHL7VNU6TYNEFJZFGR7SMXQO66UCSNORVK4XAHPGA, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.183813983 (213h53m40.054897028s) [noise=true , address=145.79.227.45:4229 , piece_id=4IHXIUBBXYN6MZYQBRBRMJZ5HYOFNCLHKOVZKFVS6M37QMRE35OA, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.186380270 (213h53m40.057463315s) [noise=true , address=145.79.227.38:4978 , piece_id=3AV4CLNBUBIXI73A3YSDKOPIMOROFBI73TVA3KNGDSBSE6KMYNEA, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.198000036 (213h53m40.069083071s) [noise=true , address=145.79.227.36:4519 , piece_id=5X7PDP7FELCD6A5566SD7YBEUN4WELNB4IDV545B27TJ6DPEIIKA, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.201922084 (213h53m40.07300513s) [noise=true , address=145.79.227.47:4267 , piece_id=FOKTCGQ7VI4GDXRJ64LDLVSALAHAL7GRFI257S6X6G2Y4IGUNW6Q, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.217629133 (213h53m40.088712178s) [noise=true , address=145.79.227.34:4905 , piece_id=GHVXBDDAG64LE2A5RRGK3POPOL3QSQGGI7UH3NIHEZTEG7A7GODA, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.217662106 (213h53m40.088745151s) [seg_pos={0 0}, upload=187397]: Segment reached optimal threshold of 13 pieces. Dec 14 13:22:40.218130388 (213h53m40.089213423s) [noise=true , address=145.79.227.44:4193 , piece_id=UMQOWDMKE3PCVS6DQUPMMRR3IYW7EUY3ABOGSFVJ3H7KGFRUZ7HA, seg_pos={0 0}, upload=187397]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8): protocol: expected piece hash; context canceled; EOF Dec 14 13:22:40.218148793 (213h53m40.089231828s) [noise=true , address=145.79.227.44:4193 , piece_id=UMQOWDMKE3PCVS6DQUPMMRR3IYW7EUY3ABOGSFVJ3H7KGFRUZ7HA, seg_pos={0 0}, upload=187397]: Long tail cancellation due to optimal threshold Dec 14 13:22:40.218265846 (213h53m40.089348891s) [noise=true , address=145.79.227.35:4481 , piece_id=LTU57UBRXVWPXJKKRNWYSIQHL7YYPNDMI7R5GARIP55AO7KS4TXQ, seg_pos={0 0}, upload=187397]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bRK8WWmAnWUwzCfHu8JtZgrruYSJCvhnsYTKcaLccJcv5RQjp): protocol: expected piece hash; context canceled; EOF Dec 14 13:22:40.218275184 (213h53m40.089358239s) [noise=true , address=145.79.227.35:4481 , piece_id=LTU57UBRXVWPXJKKRNWYSIQHL7YYPNDMI7R5GARIP55AO7KS4TXQ, seg_pos={0 0}, upload=187397]: Long tail cancellation due to optimal threshold Dec 14 13:22:40.218315581 (213h53m40.089398616s) [noise=true , address=145.79.227.42:4123 , piece_id=ES2R3XH3RQ2UAJTCWRLZBY3VHUZQVUA2EMCNBYBQSTR3N7NWU6EQ, seg_pos={0 0}, upload=187397]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM): protocol: expected piece hash; context canceled; EOF Dec 14 13:22:40.218330358 (213h53m40.089413393s) [noise=true , address=145.79.227.42:4123 , piece_id=ES2R3XH3RQ2UAJTCWRLZBY3VHUZQVUA2EMCNBYBQSTR3N7NWU6EQ, seg_pos={0 0}, upload=187397]: Long tail cancellation due to optimal threshold Dec 14 13:22:40.218347892 (213h53m40.089430937s) [seg_pos={0 0}, upload=187397]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 14 13:22:40.218452501 (213h53m40.089535556s) [upload=187397]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 14 13:22:40.218479914 (213h53m40.089562969s) [upload=187397]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 14 13:22:43.384827291 (213h53m43.255910336s) [upload=187397]: Done uploading segments... Dec 14 13:22:43.384857198 (213h53m43.255940233s) [upload=187397]: Upload finished. err: <nil> Duration: 20.148198159s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..........X...................................X.............] duration: 19.5956234s trace: https://jaeger.dev.storj.io/trace/2f31d4ad7317aa95 FAIL (2025-12-14 13:37:08 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 19.595652625s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [..............................................X.......X...X.] duration: 4.740972631s trace: https://jaeger.dev.storj.io/trace/20dc4c4ab3a6940f FAIL (2025-12-14 13:36:53 UTC) orbiter warning: context deadline exceeded Duration: 4.741002627s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...........X...................................X............] duration: 9.776387771s trace: https://jaeger.dev.storj.io/trace/2f31d4ad7317aa95 FAIL (2025-12-14 13:36:37 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 9.776415936s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [............X...................................X...........] duration: 10.373869259s trace: https://jaeger.dev.storj.io/trace/2f31d4ad7317aa95 FAIL (2025-12-14 13:36:13 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 10.373926698s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [...............................................X.......X...X] duration: 5.000773252s trace: https://jaeger.dev.storj.io/trace/20dc4c4ab3a6940f FAIL (2025-12-14 13:36:09 UTC) orbiter warning: context deadline exceeded Duration: 5.000876908s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 8.085881523s trace: https://jaeger.dev.storj.io/trace/8ca15e950587d46 FAIL (2025-12-14 13:36:03 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12YKLTah3swX14kL82eNogJxd8D8Hjfq7SPjDDQAapDnXmDctP1, Piece ID: GNFCQWHHQHCCHL7VNU6TYNEFJZFGR7SMXQO66UCSNORVK4XAHPGA) context deadline exceeded; piecestore: (Node ID: 125GxLsL5UtXkuJ1LFvF42N9iZkKtRK3rq63xdR25JQbtwoT1L5, Piece ID: AFT773CWP62ANAIFPUTDYQ35GSF6K2NFEAO7X4VKIT62CQD3NJEQ) context deadline exceeded ======LOGS====== Dec 14 13:22:35.112522281 (213h53m34.983605316s) [upload=187397]: Starting upload Dec 14 13:22:35.112587294 (213h53m34.983670329s) [upload=187397]: Done starting upload Dec 14 13:22:35.112673899 (213h53m34.983756934s) [upload=187397]: Uploading segments... Dec 14 13:22:35.112767347 (213h53m34.983850393s) [upload=187397]: Got next segment. Inline: false Dec 14 13:22:35.112800511 (213h53m34.983883556s) [upload=187397]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 14 13:22:35.112805821 (213h53m34.983888856s) [upload=187397]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 14 13:22:37.194903810 (213h53m37.065986845s) [seg_pos={0 0}, upload=187397]: Begin upload segment... Dec 14 13:22:37.195008499 (213h53m37.066091534s) [noise=true , address=145.79.227.42:4123 , piece_id=ES2R3XH3RQ2UAJTCWRLZBY3VHUZQVUA2EMCNBYBQSTR3N7NWU6EQ, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.195025751 (213h53m37.066108786s) [seg_pos={0 0}, upload=187397]: Done begin upload segment. Dec 14 13:22:37.195062162 (213h53m37.066145197s) [noise=true , address=145.79.227.44:4193 , piece_id=UMQOWDMKE3PCVS6DQUPMMRR3IYW7EUY3ABOGSFVJ3H7KGFRUZ7HA, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.195125401 (213h53m37.066208446s) [noise=true , address=145.79.227.45:4229 , piece_id=4IHXIUBBXYN6MZYQBRBRMJZ5HYOFNCLHKOVZKFVS6M37QMRE35OA, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.195764098 (213h53m37.066847133s) [noise=true , address=145.79.227.35:4482 , piece_id=RAUYG5T6DQIDOV5QM2OCE4GAPYIFBHUA3ZZR3Y6KN7ZX4OPMPEZA, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.196145995 (213h53m37.06722904s) [noise=true , address=145.79.227.47:4267 , piece_id=FOKTCGQ7VI4GDXRJ64LDLVSALAHAL7GRFI257S6X6G2Y4IGUNW6Q, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.196522081 (213h53m37.067605116s) [noise=true , address=145.79.227.36:4519 , piece_id=5X7PDP7FELCD6A5566SD7YBEUN4WELNB4IDV545B27TJ6DPEIIKA, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.196919738 (213h53m37.068002783s) [noise=true , address=145.79.227.40:4050 , piece_id=OFTY43B5JLDTXZRWXEHQZLUP6YJOL75CAL36EHSOQJR3GIHZOAJQ, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.197734449 (213h53m37.068817484s) [noise=true , address=145.79.227.28:4725 , piece_id=GNFCQWHHQHCCHL7VNU6TYNEFJZFGR7SMXQO66UCSNORVK4XAHPGA, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.197786599 (213h53m37.068869634s) [noise=true , address=145.79.227.43:4158 , piece_id=HDWGVVZAZI3R6KQFIUG74SWRF3G2USEMNKVNMKZDYISPRA74AUGQ, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.198256694 (213h53m37.069339739s) [noise=true , address=145.79.227.35:4481 , piece_id=LTU57UBRXVWPXJKKRNWYSIQHL7YYPNDMI7R5GARIP55AO7KS4TXQ, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.198675992 (213h53m37.069759027s) [noise=true , address=145.79.227.51:4411 , piece_id=V7ISMKJUJ6EDCLYOJIZBCBUUNDIWADZPZUUY7XH2F6BHVZILRXBQ, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.199111532 (213h53m37.070194567s) [noise=true , address=145.79.227.26:4655 , piece_id=NSYNQURKGVD7VTPMXJGF5RDIQKNUBLVKFYUE5CLPQ6IMIEJZQ4SQ, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.199658874 (213h53m37.07074191s) [noise=true , address=145.79.227.34:4905 , piece_id=GHVXBDDAG64LE2A5RRGK3POPOL3QSQGGI7UH3NIHEZTEG7A7GODA, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.200449229 (213h53m37.071532274s) [noise=true , address=145.79.227.38:4978 , piece_id=3AV4CLNBUBIXI73A3YSDKOPIMOROFBI73TVA3KNGDSBSE6KMYNEA, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.200877725 (213h53m37.07196077s) [noise=true , address=145.79.227.46:4601 , piece_id=AFT773CWP62ANAIFPUTDYQ35GSF6K2NFEAO7X4VKIT62CQD3NJEQ, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.207413279 (213h53m37.078496314s) [noise=true , address=145.79.227.31:4797 , piece_id=KRMTNOTWOWCIERSEUWBWZOIPDDUWYZKUBX74ZAIWV737CIKFVRLA, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:39.731884038 (213h53m39.602967073s) [upload=187397]: Next returned nil segment Dec 14 13:22:39.731901430 (213h53m39.602984475s) [upload=187397]: Waiting for error group managing segments... Dec 14 13:22:40.154962424 (213h53m40.026045469s) [noise=true , address=145.79.227.26:4655 , piece_id=NSYNQURKGVD7VTPMXJGF5RDIQKNUBLVKFYUE5CLPQ6IMIEJZQ4SQ, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.159418800 (213h53m40.030501835s) [noise=true , address=145.79.227.40:4050 , piece_id=OFTY43B5JLDTXZRWXEHQZLUP6YJOL75CAL36EHSOQJR3GIHZOAJQ, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.175663783 (213h53m40.046746829s) [noise=true , address=145.79.227.43:4158 , piece_id=HDWGVVZAZI3R6KQFIUG74SWRF3G2USEMNKVNMKZDYISPRA74AUGQ, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.175938396 (213h53m40.047021432s) [noise=true , address=145.79.227.35:4482 , piece_id=RAUYG5T6DQIDOV5QM2OCE4GAPYIFBHUA3ZZR3Y6KN7ZX4OPMPEZA, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.177599980 (213h53m40.048683025s) [noise=true , address=145.79.227.31:4797 , piece_id=KRMTNOTWOWCIERSEUWBWZOIPDDUWYZKUBX74ZAIWV737CIKFVRLA, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.181351754 (213h53m40.052434789s) [noise=true , address=145.79.227.51:4411 , piece_id=V7ISMKJUJ6EDCLYOJIZBCBUUNDIWADZPZUUY7XH2F6BHVZILRXBQ, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.181709967 (213h53m40.052793002s) [noise=true , address=145.79.227.46:4601 , piece_id=AFT773CWP62ANAIFPUTDYQ35GSF6K2NFEAO7X4VKIT62CQD3NJEQ, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.183204752 (213h53m40.054287787s) [noise=true , address=145.79.227.28:4725 , piece_id=GNFCQWHHQHCCHL7VNU6TYNEFJZFGR7SMXQO66UCSNORVK4XAHPGA, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.183813983 (213h53m40.054897028s) [noise=true , address=145.79.227.45:4229 , piece_id=4IHXIUBBXYN6MZYQBRBRMJZ5HYOFNCLHKOVZKFVS6M37QMRE35OA, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.186380270 (213h53m40.057463315s) [noise=true , address=145.79.227.38:4978 , piece_id=3AV4CLNBUBIXI73A3YSDKOPIMOROFBI73TVA3KNGDSBSE6KMYNEA, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.198000036 (213h53m40.069083071s) [noise=true , address=145.79.227.36:4519 , piece_id=5X7PDP7FELCD6A5566SD7YBEUN4WELNB4IDV545B27TJ6DPEIIKA, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.201922084 (213h53m40.07300513s) [noise=true , address=145.79.227.47:4267 , piece_id=FOKTCGQ7VI4GDXRJ64LDLVSALAHAL7GRFI257S6X6G2Y4IGUNW6Q, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.217629133 (213h53m40.088712178s) [noise=true , address=145.79.227.34:4905 , piece_id=GHVXBDDAG64LE2A5RRGK3POPOL3QSQGGI7UH3NIHEZTEG7A7GODA, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.217662106 (213h53m40.088745151s) [seg_pos={0 0}, upload=187397]: Segment reached optimal threshold of 13 pieces. Dec 14 13:22:40.218130388 (213h53m40.089213423s) [noise=true , address=145.79.227.44:4193 , piece_id=UMQOWDMKE3PCVS6DQUPMMRR3IYW7EUY3ABOGSFVJ3H7KGFRUZ7HA, seg_pos={0 0}, upload=187397]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8): protocol: expected piece hash; context canceled; EOF Dec 14 13:22:40.218148793 (213h53m40.089231828s) [noise=true , address=145.79.227.44:4193 , piece_id=UMQOWDMKE3PCVS6DQUPMMRR3IYW7EUY3ABOGSFVJ3H7KGFRUZ7HA, seg_pos={0 0}, upload=187397]: Long tail cancellation due to optimal threshold Dec 14 13:22:40.218265846 (213h53m40.089348891s) [noise=true , address=145.79.227.35:4481 , piece_id=LTU57UBRXVWPXJKKRNWYSIQHL7YYPNDMI7R5GARIP55AO7KS4TXQ, seg_pos={0 0}, upload=187397]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bRK8WWmAnWUwzCfHu8JtZgrruYSJCvhnsYTKcaLccJcv5RQjp): protocol: expected piece hash; context canceled; EOF Dec 14 13:22:40.218275184 (213h53m40.089358239s) [noise=true , address=145.79.227.35:4481 , piece_id=LTU57UBRXVWPXJKKRNWYSIQHL7YYPNDMI7R5GARIP55AO7KS4TXQ, seg_pos={0 0}, upload=187397]: Long tail cancellation due to optimal threshold Dec 14 13:22:40.218315581 (213h53m40.089398616s) [noise=true , address=145.79.227.42:4123 , piece_id=ES2R3XH3RQ2UAJTCWRLZBY3VHUZQVUA2EMCNBYBQSTR3N7NWU6EQ, seg_pos={0 0}, upload=187397]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM): protocol: expected piece hash; context canceled; EOF Dec 14 13:22:40.218330358 (213h53m40.089413393s) [noise=true , address=145.79.227.42:4123 , piece_id=ES2R3XH3RQ2UAJTCWRLZBY3VHUZQVUA2EMCNBYBQSTR3N7NWU6EQ, seg_pos={0 0}, upload=187397]: Long tail cancellation due to optimal threshold Dec 14 13:22:40.218347892 (213h53m40.089430937s) [seg_pos={0 0}, upload=187397]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 14 13:22:40.218452501 (213h53m40.089535556s) [upload=187397]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 14 13:22:40.218479914 (213h53m40.089562969s) [upload=187397]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 14 13:22:43.384827291 (213h53m43.255910336s) [upload=187397]: Done uploading segments... Dec 14 13:22:43.384857198 (213h53m43.255940233s) [upload=187397]: Upload finished. err: <nil> Duration: 8.0859273s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 8.988957641s trace: https://jaeger.dev.storj.io/trace/8ca15e950587d46 FAIL (2025-12-14 13:35:18 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12YKLTah3swX14kL82eNogJxd8D8Hjfq7SPjDDQAapDnXmDctP1, Piece ID: GNFCQWHHQHCCHL7VNU6TYNEFJZFGR7SMXQO66UCSNORVK4XAHPGA) context deadline exceeded; piecestore: (Node ID: 125GxLsL5UtXkuJ1LFvF42N9iZkKtRK3rq63xdR25JQbtwoT1L5, Piece ID: AFT773CWP62ANAIFPUTDYQ35GSF6K2NFEAO7X4VKIT62CQD3NJEQ) context deadline exceeded ======LOGS====== Dec 14 13:22:35.112522281 (213h53m34.983605316s) [upload=187397]: Starting upload Dec 14 13:22:35.112587294 (213h53m34.983670329s) [upload=187397]: Done starting upload Dec 14 13:22:35.112673899 (213h53m34.983756934s) [upload=187397]: Uploading segments... Dec 14 13:22:35.112767347 (213h53m34.983850393s) [upload=187397]: Got next segment. Inline: false Dec 14 13:22:35.112800511 (213h53m34.983883556s) [upload=187397]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 14 13:22:35.112805821 (213h53m34.983888856s) [upload=187397]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 14 13:22:37.194903810 (213h53m37.065986845s) [seg_pos={0 0}, upload=187397]: Begin upload segment... Dec 14 13:22:37.195008499 (213h53m37.066091534s) [noise=true , address=145.79.227.42:4123 , piece_id=ES2R3XH3RQ2UAJTCWRLZBY3VHUZQVUA2EMCNBYBQSTR3N7NWU6EQ, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.195025751 (213h53m37.066108786s) [seg_pos={0 0}, upload=187397]: Done begin upload segment. Dec 14 13:22:37.195062162 (213h53m37.066145197s) [noise=true , address=145.79.227.44:4193 , piece_id=UMQOWDMKE3PCVS6DQUPMMRR3IYW7EUY3ABOGSFVJ3H7KGFRUZ7HA, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.195125401 (213h53m37.066208446s) [noise=true , address=145.79.227.45:4229 , piece_id=4IHXIUBBXYN6MZYQBRBRMJZ5HYOFNCLHKOVZKFVS6M37QMRE35OA, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.195764098 (213h53m37.066847133s) [noise=true , address=145.79.227.35:4482 , piece_id=RAUYG5T6DQIDOV5QM2OCE4GAPYIFBHUA3ZZR3Y6KN7ZX4OPMPEZA, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.196145995 (213h53m37.06722904s) [noise=true , address=145.79.227.47:4267 , piece_id=FOKTCGQ7VI4GDXRJ64LDLVSALAHAL7GRFI257S6X6G2Y4IGUNW6Q, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.196522081 (213h53m37.067605116s) [noise=true , address=145.79.227.36:4519 , piece_id=5X7PDP7FELCD6A5566SD7YBEUN4WELNB4IDV545B27TJ6DPEIIKA, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.196919738 (213h53m37.068002783s) [noise=true , address=145.79.227.40:4050 , piece_id=OFTY43B5JLDTXZRWXEHQZLUP6YJOL75CAL36EHSOQJR3GIHZOAJQ, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.197734449 (213h53m37.068817484s) [noise=true , address=145.79.227.28:4725 , piece_id=GNFCQWHHQHCCHL7VNU6TYNEFJZFGR7SMXQO66UCSNORVK4XAHPGA, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.197786599 (213h53m37.068869634s) [noise=true , address=145.79.227.43:4158 , piece_id=HDWGVVZAZI3R6KQFIUG74SWRF3G2USEMNKVNMKZDYISPRA74AUGQ, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.198256694 (213h53m37.069339739s) [noise=true , address=145.79.227.35:4481 , piece_id=LTU57UBRXVWPXJKKRNWYSIQHL7YYPNDMI7R5GARIP55AO7KS4TXQ, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.198675992 (213h53m37.069759027s) [noise=true , address=145.79.227.51:4411 , piece_id=V7ISMKJUJ6EDCLYOJIZBCBUUNDIWADZPZUUY7XH2F6BHVZILRXBQ, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.199111532 (213h53m37.070194567s) [noise=true , address=145.79.227.26:4655 , piece_id=NSYNQURKGVD7VTPMXJGF5RDIQKNUBLVKFYUE5CLPQ6IMIEJZQ4SQ, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.199658874 (213h53m37.07074191s) [noise=true , address=145.79.227.34:4905 , piece_id=GHVXBDDAG64LE2A5RRGK3POPOL3QSQGGI7UH3NIHEZTEG7A7GODA, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.200449229 (213h53m37.071532274s) [noise=true , address=145.79.227.38:4978 , piece_id=3AV4CLNBUBIXI73A3YSDKOPIMOROFBI73TVA3KNGDSBSE6KMYNEA, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.200877725 (213h53m37.07196077s) [noise=true , address=145.79.227.46:4601 , piece_id=AFT773CWP62ANAIFPUTDYQ35GSF6K2NFEAO7X4VKIT62CQD3NJEQ, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.207413279 (213h53m37.078496314s) [noise=true , address=145.79.227.31:4797 , piece_id=KRMTNOTWOWCIERSEUWBWZOIPDDUWYZKUBX74ZAIWV737CIKFVRLA, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:39.731884038 (213h53m39.602967073s) [upload=187397]: Next returned nil segment Dec 14 13:22:39.731901430 (213h53m39.602984475s) [upload=187397]: Waiting for error group managing segments... Dec 14 13:22:40.154962424 (213h53m40.026045469s) [noise=true , address=145.79.227.26:4655 , piece_id=NSYNQURKGVD7VTPMXJGF5RDIQKNUBLVKFYUE5CLPQ6IMIEJZQ4SQ, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.159418800 (213h53m40.030501835s) [noise=true , address=145.79.227.40:4050 , piece_id=OFTY43B5JLDTXZRWXEHQZLUP6YJOL75CAL36EHSOQJR3GIHZOAJQ, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.175663783 (213h53m40.046746829s) [noise=true , address=145.79.227.43:4158 , piece_id=HDWGVVZAZI3R6KQFIUG74SWRF3G2USEMNKVNMKZDYISPRA74AUGQ, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.175938396 (213h53m40.047021432s) [noise=true , address=145.79.227.35:4482 , piece_id=RAUYG5T6DQIDOV5QM2OCE4GAPYIFBHUA3ZZR3Y6KN7ZX4OPMPEZA, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.177599980 (213h53m40.048683025s) [noise=true , address=145.79.227.31:4797 , piece_id=KRMTNOTWOWCIERSEUWBWZOIPDDUWYZKUBX74ZAIWV737CIKFVRLA, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.181351754 (213h53m40.052434789s) [noise=true , address=145.79.227.51:4411 , piece_id=V7ISMKJUJ6EDCLYOJIZBCBUUNDIWADZPZUUY7XH2F6BHVZILRXBQ, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.181709967 (213h53m40.052793002s) [noise=true , address=145.79.227.46:4601 , piece_id=AFT773CWP62ANAIFPUTDYQ35GSF6K2NFEAO7X4VKIT62CQD3NJEQ, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.183204752 (213h53m40.054287787s) [noise=true , address=145.79.227.28:4725 , piece_id=GNFCQWHHQHCCHL7VNU6TYNEFJZFGR7SMXQO66UCSNORVK4XAHPGA, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.183813983 (213h53m40.054897028s) [noise=true , address=145.79.227.45:4229 , piece_id=4IHXIUBBXYN6MZYQBRBRMJZ5HYOFNCLHKOVZKFVS6M37QMRE35OA, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.186380270 (213h53m40.057463315s) [noise=true , address=145.79.227.38:4978 , piece_id=3AV4CLNBUBIXI73A3YSDKOPIMOROFBI73TVA3KNGDSBSE6KMYNEA, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.198000036 (213h53m40.069083071s) [noise=true , address=145.79.227.36:4519 , piece_id=5X7PDP7FELCD6A5566SD7YBEUN4WELNB4IDV545B27TJ6DPEIIKA, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.201922084 (213h53m40.07300513s) [noise=true , address=145.79.227.47:4267 , piece_id=FOKTCGQ7VI4GDXRJ64LDLVSALAHAL7GRFI257S6X6G2Y4IGUNW6Q, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.217629133 (213h53m40.088712178s) [noise=true , address=145.79.227.34:4905 , piece_id=GHVXBDDAG64LE2A5RRGK3POPOL3QSQGGI7UH3NIHEZTEG7A7GODA, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.217662106 (213h53m40.088745151s) [seg_pos={0 0}, upload=187397]: Segment reached optimal threshold of 13 pieces. Dec 14 13:22:40.218130388 (213h53m40.089213423s) [noise=true , address=145.79.227.44:4193 , piece_id=UMQOWDMKE3PCVS6DQUPMMRR3IYW7EUY3ABOGSFVJ3H7KGFRUZ7HA, seg_pos={0 0}, upload=187397]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8): protocol: expected piece hash; context canceled; EOF Dec 14 13:22:40.218148793 (213h53m40.089231828s) [noise=true , address=145.79.227.44:4193 , piece_id=UMQOWDMKE3PCVS6DQUPMMRR3IYW7EUY3ABOGSFVJ3H7KGFRUZ7HA, seg_pos={0 0}, upload=187397]: Long tail cancellation due to optimal threshold Dec 14 13:22:40.218265846 (213h53m40.089348891s) [noise=true , address=145.79.227.35:4481 , piece_id=LTU57UBRXVWPXJKKRNWYSIQHL7YYPNDMI7R5GARIP55AO7KS4TXQ, seg_pos={0 0}, upload=187397]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bRK8WWmAnWUwzCfHu8JtZgrruYSJCvhnsYTKcaLccJcv5RQjp): protocol: expected piece hash; context canceled; EOF Dec 14 13:22:40.218275184 (213h53m40.089358239s) [noise=true , address=145.79.227.35:4481 , piece_id=LTU57UBRXVWPXJKKRNWYSIQHL7YYPNDMI7R5GARIP55AO7KS4TXQ, seg_pos={0 0}, upload=187397]: Long tail cancellation due to optimal threshold Dec 14 13:22:40.218315581 (213h53m40.089398616s) [noise=true , address=145.79.227.42:4123 , piece_id=ES2R3XH3RQ2UAJTCWRLZBY3VHUZQVUA2EMCNBYBQSTR3N7NWU6EQ, seg_pos={0 0}, upload=187397]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM): protocol: expected piece hash; context canceled; EOF Dec 14 13:22:40.218330358 (213h53m40.089413393s) [noise=true , address=145.79.227.42:4123 , piece_id=ES2R3XH3RQ2UAJTCWRLZBY3VHUZQVUA2EMCNBYBQSTR3N7NWU6EQ, seg_pos={0 0}, upload=187397]: Long tail cancellation due to optimal threshold Dec 14 13:22:40.218347892 (213h53m40.089430937s) [seg_pos={0 0}, upload=187397]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 14 13:22:40.218452501 (213h53m40.089535556s) [upload=187397]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 14 13:22:40.218479914 (213h53m40.089562969s) [upload=187397]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 14 13:22:43.384827291 (213h53m43.255910336s) [upload=187397]: Done uploading segments... Dec 14 13:22:43.384857198 (213h53m43.255940233s) [upload=187397]: Upload finished. err: <nil> Duration: 8.988992586s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [................................................X.......X...] duration: 3.625294646s trace: https://jaeger.dev.storj.io/trace/620b9f85d230c3f9 FAIL (2025-12-14 13:35:12 UTC) orbiter warning: context deadline exceeded Duration: 3.625321015s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.............X...................................X..........] duration: 9.36855082s trace: https://jaeger.dev.storj.io/trace/2f31d4ad7317aa95 FAIL (2025-12-14 13:35:06 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 9.368582029s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 11.664471326s trace: https://jaeger.dev.storj.io/trace/8ca15e950587d46 FAIL (2025-12-14 13:34:52 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12YKLTah3swX14kL82eNogJxd8D8Hjfq7SPjDDQAapDnXmDctP1, Piece ID: GNFCQWHHQHCCHL7VNU6TYNEFJZFGR7SMXQO66UCSNORVK4XAHPGA) context deadline exceeded; piecestore: (Node ID: 125GxLsL5UtXkuJ1LFvF42N9iZkKtRK3rq63xdR25JQbtwoT1L5, Piece ID: AFT773CWP62ANAIFPUTDYQ35GSF6K2NFEAO7X4VKIT62CQD3NJEQ) context deadline exceeded ======LOGS====== Dec 14 13:22:35.112522281 (213h53m34.983605316s) [upload=187397]: Starting upload Dec 14 13:22:35.112587294 (213h53m34.983670329s) [upload=187397]: Done starting upload Dec 14 13:22:35.112673899 (213h53m34.983756934s) [upload=187397]: Uploading segments... Dec 14 13:22:35.112767347 (213h53m34.983850393s) [upload=187397]: Got next segment. Inline: false Dec 14 13:22:35.112800511 (213h53m34.983883556s) [upload=187397]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 14 13:22:35.112805821 (213h53m34.983888856s) [upload=187397]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 14 13:22:37.194903810 (213h53m37.065986845s) [seg_pos={0 0}, upload=187397]: Begin upload segment... Dec 14 13:22:37.195008499 (213h53m37.066091534s) [noise=true , address=145.79.227.42:4123 , piece_id=ES2R3XH3RQ2UAJTCWRLZBY3VHUZQVUA2EMCNBYBQSTR3N7NWU6EQ, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.195025751 (213h53m37.066108786s) [seg_pos={0 0}, upload=187397]: Done begin upload segment. Dec 14 13:22:37.195062162 (213h53m37.066145197s) [noise=true , address=145.79.227.44:4193 , piece_id=UMQOWDMKE3PCVS6DQUPMMRR3IYW7EUY3ABOGSFVJ3H7KGFRUZ7HA, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.195125401 (213h53m37.066208446s) [noise=true , address=145.79.227.45:4229 , piece_id=4IHXIUBBXYN6MZYQBRBRMJZ5HYOFNCLHKOVZKFVS6M37QMRE35OA, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.195764098 (213h53m37.066847133s) [noise=true , address=145.79.227.35:4482 , piece_id=RAUYG5T6DQIDOV5QM2OCE4GAPYIFBHUA3ZZR3Y6KN7ZX4OPMPEZA, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.196145995 (213h53m37.06722904s) [noise=true , address=145.79.227.47:4267 , piece_id=FOKTCGQ7VI4GDXRJ64LDLVSALAHAL7GRFI257S6X6G2Y4IGUNW6Q, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.196522081 (213h53m37.067605116s) [noise=true , address=145.79.227.36:4519 , piece_id=5X7PDP7FELCD6A5566SD7YBEUN4WELNB4IDV545B27TJ6DPEIIKA, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.196919738 (213h53m37.068002783s) [noise=true , address=145.79.227.40:4050 , piece_id=OFTY43B5JLDTXZRWXEHQZLUP6YJOL75CAL36EHSOQJR3GIHZOAJQ, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.197734449 (213h53m37.068817484s) [noise=true , address=145.79.227.28:4725 , piece_id=GNFCQWHHQHCCHL7VNU6TYNEFJZFGR7SMXQO66UCSNORVK4XAHPGA, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.197786599 (213h53m37.068869634s) [noise=true , address=145.79.227.43:4158 , piece_id=HDWGVVZAZI3R6KQFIUG74SWRF3G2USEMNKVNMKZDYISPRA74AUGQ, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.198256694 (213h53m37.069339739s) [noise=true , address=145.79.227.35:4481 , piece_id=LTU57UBRXVWPXJKKRNWYSIQHL7YYPNDMI7R5GARIP55AO7KS4TXQ, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.198675992 (213h53m37.069759027s) [noise=true , address=145.79.227.51:4411 , piece_id=V7ISMKJUJ6EDCLYOJIZBCBUUNDIWADZPZUUY7XH2F6BHVZILRXBQ, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.199111532 (213h53m37.070194567s) [noise=true , address=145.79.227.26:4655 , piece_id=NSYNQURKGVD7VTPMXJGF5RDIQKNUBLVKFYUE5CLPQ6IMIEJZQ4SQ, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.199658874 (213h53m37.07074191s) [noise=true , address=145.79.227.34:4905 , piece_id=GHVXBDDAG64LE2A5RRGK3POPOL3QSQGGI7UH3NIHEZTEG7A7GODA, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.200449229 (213h53m37.071532274s) [noise=true , address=145.79.227.38:4978 , piece_id=3AV4CLNBUBIXI73A3YSDKOPIMOROFBI73TVA3KNGDSBSE6KMYNEA, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.200877725 (213h53m37.07196077s) [noise=true , address=145.79.227.46:4601 , piece_id=AFT773CWP62ANAIFPUTDYQ35GSF6K2NFEAO7X4VKIT62CQD3NJEQ, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.207413279 (213h53m37.078496314s) [noise=true , address=145.79.227.31:4797 , piece_id=KRMTNOTWOWCIERSEUWBWZOIPDDUWYZKUBX74ZAIWV737CIKFVRLA, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:39.731884038 (213h53m39.602967073s) [upload=187397]: Next returned nil segment Dec 14 13:22:39.731901430 (213h53m39.602984475s) [upload=187397]: Waiting for error group managing segments... Dec 14 13:22:40.154962424 (213h53m40.026045469s) [noise=true , address=145.79.227.26:4655 , piece_id=NSYNQURKGVD7VTPMXJGF5RDIQKNUBLVKFYUE5CLPQ6IMIEJZQ4SQ, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.159418800 (213h53m40.030501835s) [noise=true , address=145.79.227.40:4050 , piece_id=OFTY43B5JLDTXZRWXEHQZLUP6YJOL75CAL36EHSOQJR3GIHZOAJQ, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.175663783 (213h53m40.046746829s) [noise=true , address=145.79.227.43:4158 , piece_id=HDWGVVZAZI3R6KQFIUG74SWRF3G2USEMNKVNMKZDYISPRA74AUGQ, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.175938396 (213h53m40.047021432s) [noise=true , address=145.79.227.35:4482 , piece_id=RAUYG5T6DQIDOV5QM2OCE4GAPYIFBHUA3ZZR3Y6KN7ZX4OPMPEZA, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.177599980 (213h53m40.048683025s) [noise=true , address=145.79.227.31:4797 , piece_id=KRMTNOTWOWCIERSEUWBWZOIPDDUWYZKUBX74ZAIWV737CIKFVRLA, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.181351754 (213h53m40.052434789s) [noise=true , address=145.79.227.51:4411 , piece_id=V7ISMKJUJ6EDCLYOJIZBCBUUNDIWADZPZUUY7XH2F6BHVZILRXBQ, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.181709967 (213h53m40.052793002s) [noise=true , address=145.79.227.46:4601 , piece_id=AFT773CWP62ANAIFPUTDYQ35GSF6K2NFEAO7X4VKIT62CQD3NJEQ, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.183204752 (213h53m40.054287787s) [noise=true , address=145.79.227.28:4725 , piece_id=GNFCQWHHQHCCHL7VNU6TYNEFJZFGR7SMXQO66UCSNORVK4XAHPGA, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.183813983 (213h53m40.054897028s) [noise=true , address=145.79.227.45:4229 , piece_id=4IHXIUBBXYN6MZYQBRBRMJZ5HYOFNCLHKOVZKFVS6M37QMRE35OA, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.186380270 (213h53m40.057463315s) [noise=true , address=145.79.227.38:4978 , piece_id=3AV4CLNBUBIXI73A3YSDKOPIMOROFBI73TVA3KNGDSBSE6KMYNEA, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.198000036 (213h53m40.069083071s) [noise=true , address=145.79.227.36:4519 , piece_id=5X7PDP7FELCD6A5566SD7YBEUN4WELNB4IDV545B27TJ6DPEIIKA, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.201922084 (213h53m40.07300513s) [noise=true , address=145.79.227.47:4267 , piece_id=FOKTCGQ7VI4GDXRJ64LDLVSALAHAL7GRFI257S6X6G2Y4IGUNW6Q, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.217629133 (213h53m40.088712178s) [noise=true , address=145.79.227.34:4905 , piece_id=GHVXBDDAG64LE2A5RRGK3POPOL3QSQGGI7UH3NIHEZTEG7A7GODA, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.217662106 (213h53m40.088745151s) [seg_pos={0 0}, upload=187397]: Segment reached optimal threshold of 13 pieces. Dec 14 13:22:40.218130388 (213h53m40.089213423s) [noise=true , address=145.79.227.44:4193 , piece_id=UMQOWDMKE3PCVS6DQUPMMRR3IYW7EUY3ABOGSFVJ3H7KGFRUZ7HA, seg_pos={0 0}, upload=187397]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8): protocol: expected piece hash; context canceled; EOF Dec 14 13:22:40.218148793 (213h53m40.089231828s) [noise=true , address=145.79.227.44:4193 , piece_id=UMQOWDMKE3PCVS6DQUPMMRR3IYW7EUY3ABOGSFVJ3H7KGFRUZ7HA, seg_pos={0 0}, upload=187397]: Long tail cancellation due to optimal threshold Dec 14 13:22:40.218265846 (213h53m40.089348891s) [noise=true , address=145.79.227.35:4481 , piece_id=LTU57UBRXVWPXJKKRNWYSIQHL7YYPNDMI7R5GARIP55AO7KS4TXQ, seg_pos={0 0}, upload=187397]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bRK8WWmAnWUwzCfHu8JtZgrruYSJCvhnsYTKcaLccJcv5RQjp): protocol: expected piece hash; context canceled; EOF Dec 14 13:22:40.218275184 (213h53m40.089358239s) [noise=true , address=145.79.227.35:4481 , piece_id=LTU57UBRXVWPXJKKRNWYSIQHL7YYPNDMI7R5GARIP55AO7KS4TXQ, seg_pos={0 0}, upload=187397]: Long tail cancellation due to optimal threshold Dec 14 13:22:40.218315581 (213h53m40.089398616s) [noise=true , address=145.79.227.42:4123 , piece_id=ES2R3XH3RQ2UAJTCWRLZBY3VHUZQVUA2EMCNBYBQSTR3N7NWU6EQ, seg_pos={0 0}, upload=187397]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM): protocol: expected piece hash; context canceled; EOF Dec 14 13:22:40.218330358 (213h53m40.089413393s) [noise=true , address=145.79.227.42:4123 , piece_id=ES2R3XH3RQ2UAJTCWRLZBY3VHUZQVUA2EMCNBYBQSTR3N7NWU6EQ, seg_pos={0 0}, upload=187397]: Long tail cancellation due to optimal threshold Dec 14 13:22:40.218347892 (213h53m40.089430937s) [seg_pos={0 0}, upload=187397]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 14 13:22:40.218452501 (213h53m40.089535556s) [upload=187397]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 14 13:22:40.218479914 (213h53m40.089562969s) [upload=187397]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 14 13:22:43.384827291 (213h53m43.255910336s) [upload=187397]: Done uploading segments... Dec 14 13:22:43.384857198 (213h53m43.255940233s) [upload=187397]: Upload finished. err: <nil> Duration: 11.664505129s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [.................................................X.......X..] duration: 3.657411513s trace: https://jaeger.dev.storj.io/trace/620b9f85d230c3f9 FAIL (2025-12-14 13:34:34 UTC) orbiter warning: context deadline exceeded Duration: 3.657439227s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 9.333434701s trace: https://jaeger.dev.storj.io/trace/8ca15e950587d46 FAIL (2025-12-14 13:34:25 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12YKLTah3swX14kL82eNogJxd8D8Hjfq7SPjDDQAapDnXmDctP1, Piece ID: GNFCQWHHQHCCHL7VNU6TYNEFJZFGR7SMXQO66UCSNORVK4XAHPGA) context deadline exceeded; piecestore: (Node ID: 125GxLsL5UtXkuJ1LFvF42N9iZkKtRK3rq63xdR25JQbtwoT1L5, Piece ID: AFT773CWP62ANAIFPUTDYQ35GSF6K2NFEAO7X4VKIT62CQD3NJEQ) context deadline exceeded ======LOGS====== Dec 14 13:22:35.112522281 (213h53m34.983605316s) [upload=187397]: Starting upload Dec 14 13:22:35.112587294 (213h53m34.983670329s) [upload=187397]: Done starting upload Dec 14 13:22:35.112673899 (213h53m34.983756934s) [upload=187397]: Uploading segments... Dec 14 13:22:35.112767347 (213h53m34.983850393s) [upload=187397]: Got next segment. Inline: false Dec 14 13:22:35.112800511 (213h53m34.983883556s) [upload=187397]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 14 13:22:35.112805821 (213h53m34.983888856s) [upload=187397]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 14 13:22:37.194903810 (213h53m37.065986845s) [seg_pos={0 0}, upload=187397]: Begin upload segment... Dec 14 13:22:37.195008499 (213h53m37.066091534s) [noise=true , address=145.79.227.42:4123 , piece_id=ES2R3XH3RQ2UAJTCWRLZBY3VHUZQVUA2EMCNBYBQSTR3N7NWU6EQ, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.195025751 (213h53m37.066108786s) [seg_pos={0 0}, upload=187397]: Done begin upload segment. Dec 14 13:22:37.195062162 (213h53m37.066145197s) [noise=true , address=145.79.227.44:4193 , piece_id=UMQOWDMKE3PCVS6DQUPMMRR3IYW7EUY3ABOGSFVJ3H7KGFRUZ7HA, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.195125401 (213h53m37.066208446s) [noise=true , address=145.79.227.45:4229 , piece_id=4IHXIUBBXYN6MZYQBRBRMJZ5HYOFNCLHKOVZKFVS6M37QMRE35OA, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.195764098 (213h53m37.066847133s) [noise=true , address=145.79.227.35:4482 , piece_id=RAUYG5T6DQIDOV5QM2OCE4GAPYIFBHUA3ZZR3Y6KN7ZX4OPMPEZA, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.196145995 (213h53m37.06722904s) [noise=true , address=145.79.227.47:4267 , piece_id=FOKTCGQ7VI4GDXRJ64LDLVSALAHAL7GRFI257S6X6G2Y4IGUNW6Q, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.196522081 (213h53m37.067605116s) [noise=true , address=145.79.227.36:4519 , piece_id=5X7PDP7FELCD6A5566SD7YBEUN4WELNB4IDV545B27TJ6DPEIIKA, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.196919738 (213h53m37.068002783s) [noise=true , address=145.79.227.40:4050 , piece_id=OFTY43B5JLDTXZRWXEHQZLUP6YJOL75CAL36EHSOQJR3GIHZOAJQ, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.197734449 (213h53m37.068817484s) [noise=true , address=145.79.227.28:4725 , piece_id=GNFCQWHHQHCCHL7VNU6TYNEFJZFGR7SMXQO66UCSNORVK4XAHPGA, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.197786599 (213h53m37.068869634s) [noise=true , address=145.79.227.43:4158 , piece_id=HDWGVVZAZI3R6KQFIUG74SWRF3G2USEMNKVNMKZDYISPRA74AUGQ, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.198256694 (213h53m37.069339739s) [noise=true , address=145.79.227.35:4481 , piece_id=LTU57UBRXVWPXJKKRNWYSIQHL7YYPNDMI7R5GARIP55AO7KS4TXQ, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.198675992 (213h53m37.069759027s) [noise=true , address=145.79.227.51:4411 , piece_id=V7ISMKJUJ6EDCLYOJIZBCBUUNDIWADZPZUUY7XH2F6BHVZILRXBQ, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.199111532 (213h53m37.070194567s) [noise=true , address=145.79.227.26:4655 , piece_id=NSYNQURKGVD7VTPMXJGF5RDIQKNUBLVKFYUE5CLPQ6IMIEJZQ4SQ, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.199658874 (213h53m37.07074191s) [noise=true , address=145.79.227.34:4905 , piece_id=GHVXBDDAG64LE2A5RRGK3POPOL3QSQGGI7UH3NIHEZTEG7A7GODA, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.200449229 (213h53m37.071532274s) [noise=true , address=145.79.227.38:4978 , piece_id=3AV4CLNBUBIXI73A3YSDKOPIMOROFBI73TVA3KNGDSBSE6KMYNEA, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.200877725 (213h53m37.07196077s) [noise=true , address=145.79.227.46:4601 , piece_id=AFT773CWP62ANAIFPUTDYQ35GSF6K2NFEAO7X4VKIT62CQD3NJEQ, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.207413279 (213h53m37.078496314s) [noise=true , address=145.79.227.31:4797 , piece_id=KRMTNOTWOWCIERSEUWBWZOIPDDUWYZKUBX74ZAIWV737CIKFVRLA, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:39.731884038 (213h53m39.602967073s) [upload=187397]: Next returned nil segment Dec 14 13:22:39.731901430 (213h53m39.602984475s) [upload=187397]: Waiting for error group managing segments... Dec 14 13:22:40.154962424 (213h53m40.026045469s) [noise=true , address=145.79.227.26:4655 , piece_id=NSYNQURKGVD7VTPMXJGF5RDIQKNUBLVKFYUE5CLPQ6IMIEJZQ4SQ, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.159418800 (213h53m40.030501835s) [noise=true , address=145.79.227.40:4050 , piece_id=OFTY43B5JLDTXZRWXEHQZLUP6YJOL75CAL36EHSOQJR3GIHZOAJQ, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.175663783 (213h53m40.046746829s) [noise=true , address=145.79.227.43:4158 , piece_id=HDWGVVZAZI3R6KQFIUG74SWRF3G2USEMNKVNMKZDYISPRA74AUGQ, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.175938396 (213h53m40.047021432s) [noise=true , address=145.79.227.35:4482 , piece_id=RAUYG5T6DQIDOV5QM2OCE4GAPYIFBHUA3ZZR3Y6KN7ZX4OPMPEZA, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.177599980 (213h53m40.048683025s) [noise=true , address=145.79.227.31:4797 , piece_id=KRMTNOTWOWCIERSEUWBWZOIPDDUWYZKUBX74ZAIWV737CIKFVRLA, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.181351754 (213h53m40.052434789s) [noise=true , address=145.79.227.51:4411 , piece_id=V7ISMKJUJ6EDCLYOJIZBCBUUNDIWADZPZUUY7XH2F6BHVZILRXBQ, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.181709967 (213h53m40.052793002s) [noise=true , address=145.79.227.46:4601 , piece_id=AFT773CWP62ANAIFPUTDYQ35GSF6K2NFEAO7X4VKIT62CQD3NJEQ, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.183204752 (213h53m40.054287787s) [noise=true , address=145.79.227.28:4725 , piece_id=GNFCQWHHQHCCHL7VNU6TYNEFJZFGR7SMXQO66UCSNORVK4XAHPGA, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.183813983 (213h53m40.054897028s) [noise=true , address=145.79.227.45:4229 , piece_id=4IHXIUBBXYN6MZYQBRBRMJZ5HYOFNCLHKOVZKFVS6M37QMRE35OA, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.186380270 (213h53m40.057463315s) [noise=true , address=145.79.227.38:4978 , piece_id=3AV4CLNBUBIXI73A3YSDKOPIMOROFBI73TVA3KNGDSBSE6KMYNEA, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.198000036 (213h53m40.069083071s) [noise=true , address=145.79.227.36:4519 , piece_id=5X7PDP7FELCD6A5566SD7YBEUN4WELNB4IDV545B27TJ6DPEIIKA, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.201922084 (213h53m40.07300513s) [noise=true , address=145.79.227.47:4267 , piece_id=FOKTCGQ7VI4GDXRJ64LDLVSALAHAL7GRFI257S6X6G2Y4IGUNW6Q, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.217629133 (213h53m40.088712178s) [noise=true , address=145.79.227.34:4905 , piece_id=GHVXBDDAG64LE2A5RRGK3POPOL3QSQGGI7UH3NIHEZTEG7A7GODA, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.217662106 (213h53m40.088745151s) [seg_pos={0 0}, upload=187397]: Segment reached optimal threshold of 13 pieces. Dec 14 13:22:40.218130388 (213h53m40.089213423s) [noise=true , address=145.79.227.44:4193 , piece_id=UMQOWDMKE3PCVS6DQUPMMRR3IYW7EUY3ABOGSFVJ3H7KGFRUZ7HA, seg_pos={0 0}, upload=187397]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8): protocol: expected piece hash; context canceled; EOF Dec 14 13:22:40.218148793 (213h53m40.089231828s) [noise=true , address=145.79.227.44:4193 , piece_id=UMQOWDMKE3PCVS6DQUPMMRR3IYW7EUY3ABOGSFVJ3H7KGFRUZ7HA, seg_pos={0 0}, upload=187397]: Long tail cancellation due to optimal threshold Dec 14 13:22:40.218265846 (213h53m40.089348891s) [noise=true , address=145.79.227.35:4481 , piece_id=LTU57UBRXVWPXJKKRNWYSIQHL7YYPNDMI7R5GARIP55AO7KS4TXQ, seg_pos={0 0}, upload=187397]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bRK8WWmAnWUwzCfHu8JtZgrruYSJCvhnsYTKcaLccJcv5RQjp): protocol: expected piece hash; context canceled; EOF Dec 14 13:22:40.218275184 (213h53m40.089358239s) [noise=true , address=145.79.227.35:4481 , piece_id=LTU57UBRXVWPXJKKRNWYSIQHL7YYPNDMI7R5GARIP55AO7KS4TXQ, seg_pos={0 0}, upload=187397]: Long tail cancellation due to optimal threshold Dec 14 13:22:40.218315581 (213h53m40.089398616s) [noise=true , address=145.79.227.42:4123 , piece_id=ES2R3XH3RQ2UAJTCWRLZBY3VHUZQVUA2EMCNBYBQSTR3N7NWU6EQ, seg_pos={0 0}, upload=187397]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM): protocol: expected piece hash; context canceled; EOF Dec 14 13:22:40.218330358 (213h53m40.089413393s) [noise=true , address=145.79.227.42:4123 , piece_id=ES2R3XH3RQ2UAJTCWRLZBY3VHUZQVUA2EMCNBYBQSTR3N7NWU6EQ, seg_pos={0 0}, upload=187397]: Long tail cancellation due to optimal threshold Dec 14 13:22:40.218347892 (213h53m40.089430937s) [seg_pos={0 0}, upload=187397]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 14 13:22:40.218452501 (213h53m40.089535556s) [upload=187397]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 14 13:22:40.218479914 (213h53m40.089562969s) [upload=187397]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 14 13:22:43.384827291 (213h53m43.255910336s) [upload=187397]: Done uploading segments... Dec 14 13:22:43.384857198 (213h53m43.255940233s) [upload=187397]: Upload finished. err: <nil> Duration: 9.333461872s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..............X...................................X.........] duration: 24.750023978s trace: https://jaeger.dev.storj.io/trace/2f31d4ad7317aa95 FAIL (2025-12-14 13:34:18 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 24.750116915s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 18.199949144s trace: https://jaeger.dev.storj.io/trace/8ca15e950587d46 FAIL (2025-12-14 13:34:15 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12YKLTah3swX14kL82eNogJxd8D8Hjfq7SPjDDQAapDnXmDctP1, Piece ID: GNFCQWHHQHCCHL7VNU6TYNEFJZFGR7SMXQO66UCSNORVK4XAHPGA) context deadline exceeded; piecestore: (Node ID: 125GxLsL5UtXkuJ1LFvF42N9iZkKtRK3rq63xdR25JQbtwoT1L5, Piece ID: AFT773CWP62ANAIFPUTDYQ35GSF6K2NFEAO7X4VKIT62CQD3NJEQ) context deadline exceeded ======LOGS====== Dec 14 13:22:35.112522281 (213h53m34.983605316s) [upload=187397]: Starting upload Dec 14 13:22:35.112587294 (213h53m34.983670329s) [upload=187397]: Done starting upload Dec 14 13:22:35.112673899 (213h53m34.983756934s) [upload=187397]: Uploading segments... Dec 14 13:22:35.112767347 (213h53m34.983850393s) [upload=187397]: Got next segment. Inline: false Dec 14 13:22:35.112800511 (213h53m34.983883556s) [upload=187397]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 14 13:22:35.112805821 (213h53m34.983888856s) [upload=187397]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 14 13:22:37.194903810 (213h53m37.065986845s) [seg_pos={0 0}, upload=187397]: Begin upload segment... Dec 14 13:22:37.195008499 (213h53m37.066091534s) [noise=true , address=145.79.227.42:4123 , piece_id=ES2R3XH3RQ2UAJTCWRLZBY3VHUZQVUA2EMCNBYBQSTR3N7NWU6EQ, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.195025751 (213h53m37.066108786s) [seg_pos={0 0}, upload=187397]: Done begin upload segment. Dec 14 13:22:37.195062162 (213h53m37.066145197s) [noise=true , address=145.79.227.44:4193 , piece_id=UMQOWDMKE3PCVS6DQUPMMRR3IYW7EUY3ABOGSFVJ3H7KGFRUZ7HA, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.195125401 (213h53m37.066208446s) [noise=true , address=145.79.227.45:4229 , piece_id=4IHXIUBBXYN6MZYQBRBRMJZ5HYOFNCLHKOVZKFVS6M37QMRE35OA, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.195764098 (213h53m37.066847133s) [noise=true , address=145.79.227.35:4482 , piece_id=RAUYG5T6DQIDOV5QM2OCE4GAPYIFBHUA3ZZR3Y6KN7ZX4OPMPEZA, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.196145995 (213h53m37.06722904s) [noise=true , address=145.79.227.47:4267 , piece_id=FOKTCGQ7VI4GDXRJ64LDLVSALAHAL7GRFI257S6X6G2Y4IGUNW6Q, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.196522081 (213h53m37.067605116s) [noise=true , address=145.79.227.36:4519 , piece_id=5X7PDP7FELCD6A5566SD7YBEUN4WELNB4IDV545B27TJ6DPEIIKA, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.196919738 (213h53m37.068002783s) [noise=true , address=145.79.227.40:4050 , piece_id=OFTY43B5JLDTXZRWXEHQZLUP6YJOL75CAL36EHSOQJR3GIHZOAJQ, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.197734449 (213h53m37.068817484s) [noise=true , address=145.79.227.28:4725 , piece_id=GNFCQWHHQHCCHL7VNU6TYNEFJZFGR7SMXQO66UCSNORVK4XAHPGA, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.197786599 (213h53m37.068869634s) [noise=true , address=145.79.227.43:4158 , piece_id=HDWGVVZAZI3R6KQFIUG74SWRF3G2USEMNKVNMKZDYISPRA74AUGQ, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.198256694 (213h53m37.069339739s) [noise=true , address=145.79.227.35:4481 , piece_id=LTU57UBRXVWPXJKKRNWYSIQHL7YYPNDMI7R5GARIP55AO7KS4TXQ, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.198675992 (213h53m37.069759027s) [noise=true , address=145.79.227.51:4411 , piece_id=V7ISMKJUJ6EDCLYOJIZBCBUUNDIWADZPZUUY7XH2F6BHVZILRXBQ, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.199111532 (213h53m37.070194567s) [noise=true , address=145.79.227.26:4655 , piece_id=NSYNQURKGVD7VTPMXJGF5RDIQKNUBLVKFYUE5CLPQ6IMIEJZQ4SQ, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.199658874 (213h53m37.07074191s) [noise=true , address=145.79.227.34:4905 , piece_id=GHVXBDDAG64LE2A5RRGK3POPOL3QSQGGI7UH3NIHEZTEG7A7GODA, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.200449229 (213h53m37.071532274s) [noise=true , address=145.79.227.38:4978 , piece_id=3AV4CLNBUBIXI73A3YSDKOPIMOROFBI73TVA3KNGDSBSE6KMYNEA, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.200877725 (213h53m37.07196077s) [noise=true , address=145.79.227.46:4601 , piece_id=AFT773CWP62ANAIFPUTDYQ35GSF6K2NFEAO7X4VKIT62CQD3NJEQ, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.207413279 (213h53m37.078496314s) [noise=true , address=145.79.227.31:4797 , piece_id=KRMTNOTWOWCIERSEUWBWZOIPDDUWYZKUBX74ZAIWV737CIKFVRLA, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:39.731884038 (213h53m39.602967073s) [upload=187397]: Next returned nil segment Dec 14 13:22:39.731901430 (213h53m39.602984475s) [upload=187397]: Waiting for error group managing segments... Dec 14 13:22:40.154962424 (213h53m40.026045469s) [noise=true , address=145.79.227.26:4655 , piece_id=NSYNQURKGVD7VTPMXJGF5RDIQKNUBLVKFYUE5CLPQ6IMIEJZQ4SQ, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.159418800 (213h53m40.030501835s) [noise=true , address=145.79.227.40:4050 , piece_id=OFTY43B5JLDTXZRWXEHQZLUP6YJOL75CAL36EHSOQJR3GIHZOAJQ, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.175663783 (213h53m40.046746829s) [noise=true , address=145.79.227.43:4158 , piece_id=HDWGVVZAZI3R6KQFIUG74SWRF3G2USEMNKVNMKZDYISPRA74AUGQ, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.175938396 (213h53m40.047021432s) [noise=true , address=145.79.227.35:4482 , piece_id=RAUYG5T6DQIDOV5QM2OCE4GAPYIFBHUA3ZZR3Y6KN7ZX4OPMPEZA, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.177599980 (213h53m40.048683025s) [noise=true , address=145.79.227.31:4797 , piece_id=KRMTNOTWOWCIERSEUWBWZOIPDDUWYZKUBX74ZAIWV737CIKFVRLA, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.181351754 (213h53m40.052434789s) [noise=true , address=145.79.227.51:4411 , piece_id=V7ISMKJUJ6EDCLYOJIZBCBUUNDIWADZPZUUY7XH2F6BHVZILRXBQ, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.181709967 (213h53m40.052793002s) [noise=true , address=145.79.227.46:4601 , piece_id=AFT773CWP62ANAIFPUTDYQ35GSF6K2NFEAO7X4VKIT62CQD3NJEQ, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.183204752 (213h53m40.054287787s) [noise=true , address=145.79.227.28:4725 , piece_id=GNFCQWHHQHCCHL7VNU6TYNEFJZFGR7SMXQO66UCSNORVK4XAHPGA, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.183813983 (213h53m40.054897028s) [noise=true , address=145.79.227.45:4229 , piece_id=4IHXIUBBXYN6MZYQBRBRMJZ5HYOFNCLHKOVZKFVS6M37QMRE35OA, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.186380270 (213h53m40.057463315s) [noise=true , address=145.79.227.38:4978 , piece_id=3AV4CLNBUBIXI73A3YSDKOPIMOROFBI73TVA3KNGDSBSE6KMYNEA, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.198000036 (213h53m40.069083071s) [noise=true , address=145.79.227.36:4519 , piece_id=5X7PDP7FELCD6A5566SD7YBEUN4WELNB4IDV545B27TJ6DPEIIKA, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.201922084 (213h53m40.07300513s) [noise=true , address=145.79.227.47:4267 , piece_id=FOKTCGQ7VI4GDXRJ64LDLVSALAHAL7GRFI257S6X6G2Y4IGUNW6Q, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.217629133 (213h53m40.088712178s) [noise=true , address=145.79.227.34:4905 , piece_id=GHVXBDDAG64LE2A5RRGK3POPOL3QSQGGI7UH3NIHEZTEG7A7GODA, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.217662106 (213h53m40.088745151s) [seg_pos={0 0}, upload=187397]: Segment reached optimal threshold of 13 pieces. Dec 14 13:22:40.218130388 (213h53m40.089213423s) [noise=true , address=145.79.227.44:4193 , piece_id=UMQOWDMKE3PCVS6DQUPMMRR3IYW7EUY3ABOGSFVJ3H7KGFRUZ7HA, seg_pos={0 0}, upload=187397]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8): protocol: expected piece hash; context canceled; EOF Dec 14 13:22:40.218148793 (213h53m40.089231828s) [noise=true , address=145.79.227.44:4193 , piece_id=UMQOWDMKE3PCVS6DQUPMMRR3IYW7EUY3ABOGSFVJ3H7KGFRUZ7HA, seg_pos={0 0}, upload=187397]: Long tail cancellation due to optimal threshold Dec 14 13:22:40.218265846 (213h53m40.089348891s) [noise=true , address=145.79.227.35:4481 , piece_id=LTU57UBRXVWPXJKKRNWYSIQHL7YYPNDMI7R5GARIP55AO7KS4TXQ, seg_pos={0 0}, upload=187397]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bRK8WWmAnWUwzCfHu8JtZgrruYSJCvhnsYTKcaLccJcv5RQjp): protocol: expected piece hash; context canceled; EOF Dec 14 13:22:40.218275184 (213h53m40.089358239s) [noise=true , address=145.79.227.35:4481 , piece_id=LTU57UBRXVWPXJKKRNWYSIQHL7YYPNDMI7R5GARIP55AO7KS4TXQ, seg_pos={0 0}, upload=187397]: Long tail cancellation due to optimal threshold Dec 14 13:22:40.218315581 (213h53m40.089398616s) [noise=true , address=145.79.227.42:4123 , piece_id=ES2R3XH3RQ2UAJTCWRLZBY3VHUZQVUA2EMCNBYBQSTR3N7NWU6EQ, seg_pos={0 0}, upload=187397]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM): protocol: expected piece hash; context canceled; EOF Dec 14 13:22:40.218330358 (213h53m40.089413393s) [noise=true , address=145.79.227.42:4123 , piece_id=ES2R3XH3RQ2UAJTCWRLZBY3VHUZQVUA2EMCNBYBQSTR3N7NWU6EQ, seg_pos={0 0}, upload=187397]: Long tail cancellation due to optimal threshold Dec 14 13:22:40.218347892 (213h53m40.089430937s) [seg_pos={0 0}, upload=187397]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 14 13:22:40.218452501 (213h53m40.089535556s) [upload=187397]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 14 13:22:40.218479914 (213h53m40.089562969s) [upload=187397]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 14 13:22:43.384827291 (213h53m43.255910336s) [upload=187397]: Done uploading segments... Dec 14 13:22:43.384857198 (213h53m43.255940233s) [upload=187397]: Upload finished. err: <nil> Duration: 18.199974021s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [..................................................X.......X.] duration: 4.727150548s trace: https://jaeger.dev.storj.io/trace/620b9f85d230c3f9 FAIL (2025-12-14 13:33:53 UTC) orbiter warning: context deadline exceeded Duration: 4.727193759s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...............X...................................X........] duration: 27.301229585s trace: https://jaeger.dev.storj.io/trace/2f31d4ad7317aa95 FAIL (2025-12-14 13:33:30 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 27.301324817s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [...................................................X.......X] duration: 5.000755044s trace: https://jaeger.dev.storj.io/trace/620b9f85d230c3f9 FAIL (2025-12-14 13:33:20 UTC) orbiter warning: context deadline exceeded Duration: 5.000779963s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 9.253193946s trace: https://jaeger.dev.storj.io/trace/8ca15e950587d46 FAIL (2025-12-14 13:32:55 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12YKLTah3swX14kL82eNogJxd8D8Hjfq7SPjDDQAapDnXmDctP1, Piece ID: GNFCQWHHQHCCHL7VNU6TYNEFJZFGR7SMXQO66UCSNORVK4XAHPGA) context deadline exceeded; piecestore: (Node ID: 125GxLsL5UtXkuJ1LFvF42N9iZkKtRK3rq63xdR25JQbtwoT1L5, Piece ID: AFT773CWP62ANAIFPUTDYQ35GSF6K2NFEAO7X4VKIT62CQD3NJEQ) context deadline exceeded ======LOGS====== Dec 14 13:22:35.112522281 (213h53m34.983605316s) [upload=187397]: Starting upload Dec 14 13:22:35.112587294 (213h53m34.983670329s) [upload=187397]: Done starting upload Dec 14 13:22:35.112673899 (213h53m34.983756934s) [upload=187397]: Uploading segments... Dec 14 13:22:35.112767347 (213h53m34.983850393s) [upload=187397]: Got next segment. Inline: false Dec 14 13:22:35.112800511 (213h53m34.983883556s) [upload=187397]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 14 13:22:35.112805821 (213h53m34.983888856s) [upload=187397]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 14 13:22:37.194903810 (213h53m37.065986845s) [seg_pos={0 0}, upload=187397]: Begin upload segment... Dec 14 13:22:37.195008499 (213h53m37.066091534s) [noise=true , address=145.79.227.42:4123 , piece_id=ES2R3XH3RQ2UAJTCWRLZBY3VHUZQVUA2EMCNBYBQSTR3N7NWU6EQ, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.195025751 (213h53m37.066108786s) [seg_pos={0 0}, upload=187397]: Done begin upload segment. Dec 14 13:22:37.195062162 (213h53m37.066145197s) [noise=true , address=145.79.227.44:4193 , piece_id=UMQOWDMKE3PCVS6DQUPMMRR3IYW7EUY3ABOGSFVJ3H7KGFRUZ7HA, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.195125401 (213h53m37.066208446s) [noise=true , address=145.79.227.45:4229 , piece_id=4IHXIUBBXYN6MZYQBRBRMJZ5HYOFNCLHKOVZKFVS6M37QMRE35OA, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.195764098 (213h53m37.066847133s) [noise=true , address=145.79.227.35:4482 , piece_id=RAUYG5T6DQIDOV5QM2OCE4GAPYIFBHUA3ZZR3Y6KN7ZX4OPMPEZA, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.196145995 (213h53m37.06722904s) [noise=true , address=145.79.227.47:4267 , piece_id=FOKTCGQ7VI4GDXRJ64LDLVSALAHAL7GRFI257S6X6G2Y4IGUNW6Q, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.196522081 (213h53m37.067605116s) [noise=true , address=145.79.227.36:4519 , piece_id=5X7PDP7FELCD6A5566SD7YBEUN4WELNB4IDV545B27TJ6DPEIIKA, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.196919738 (213h53m37.068002783s) [noise=true , address=145.79.227.40:4050 , piece_id=OFTY43B5JLDTXZRWXEHQZLUP6YJOL75CAL36EHSOQJR3GIHZOAJQ, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.197734449 (213h53m37.068817484s) [noise=true , address=145.79.227.28:4725 , piece_id=GNFCQWHHQHCCHL7VNU6TYNEFJZFGR7SMXQO66UCSNORVK4XAHPGA, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.197786599 (213h53m37.068869634s) [noise=true , address=145.79.227.43:4158 , piece_id=HDWGVVZAZI3R6KQFIUG74SWRF3G2USEMNKVNMKZDYISPRA74AUGQ, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.198256694 (213h53m37.069339739s) [noise=true , address=145.79.227.35:4481 , piece_id=LTU57UBRXVWPXJKKRNWYSIQHL7YYPNDMI7R5GARIP55AO7KS4TXQ, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.198675992 (213h53m37.069759027s) [noise=true , address=145.79.227.51:4411 , piece_id=V7ISMKJUJ6EDCLYOJIZBCBUUNDIWADZPZUUY7XH2F6BHVZILRXBQ, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.199111532 (213h53m37.070194567s) [noise=true , address=145.79.227.26:4655 , piece_id=NSYNQURKGVD7VTPMXJGF5RDIQKNUBLVKFYUE5CLPQ6IMIEJZQ4SQ, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.199658874 (213h53m37.07074191s) [noise=true , address=145.79.227.34:4905 , piece_id=GHVXBDDAG64LE2A5RRGK3POPOL3QSQGGI7UH3NIHEZTEG7A7GODA, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.200449229 (213h53m37.071532274s) [noise=true , address=145.79.227.38:4978 , piece_id=3AV4CLNBUBIXI73A3YSDKOPIMOROFBI73TVA3KNGDSBSE6KMYNEA, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.200877725 (213h53m37.07196077s) [noise=true , address=145.79.227.46:4601 , piece_id=AFT773CWP62ANAIFPUTDYQ35GSF6K2NFEAO7X4VKIT62CQD3NJEQ, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.207413279 (213h53m37.078496314s) [noise=true , address=145.79.227.31:4797 , piece_id=KRMTNOTWOWCIERSEUWBWZOIPDDUWYZKUBX74ZAIWV737CIKFVRLA, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:39.731884038 (213h53m39.602967073s) [upload=187397]: Next returned nil segment Dec 14 13:22:39.731901430 (213h53m39.602984475s) [upload=187397]: Waiting for error group managing segments... Dec 14 13:22:40.154962424 (213h53m40.026045469s) [noise=true , address=145.79.227.26:4655 , piece_id=NSYNQURKGVD7VTPMXJGF5RDIQKNUBLVKFYUE5CLPQ6IMIEJZQ4SQ, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.159418800 (213h53m40.030501835s) [noise=true , address=145.79.227.40:4050 , piece_id=OFTY43B5JLDTXZRWXEHQZLUP6YJOL75CAL36EHSOQJR3GIHZOAJQ, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.175663783 (213h53m40.046746829s) [noise=true , address=145.79.227.43:4158 , piece_id=HDWGVVZAZI3R6KQFIUG74SWRF3G2USEMNKVNMKZDYISPRA74AUGQ, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.175938396 (213h53m40.047021432s) [noise=true , address=145.79.227.35:4482 , piece_id=RAUYG5T6DQIDOV5QM2OCE4GAPYIFBHUA3ZZR3Y6KN7ZX4OPMPEZA, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.177599980 (213h53m40.048683025s) [noise=true , address=145.79.227.31:4797 , piece_id=KRMTNOTWOWCIERSEUWBWZOIPDDUWYZKUBX74ZAIWV737CIKFVRLA, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.181351754 (213h53m40.052434789s) [noise=true , address=145.79.227.51:4411 , piece_id=V7ISMKJUJ6EDCLYOJIZBCBUUNDIWADZPZUUY7XH2F6BHVZILRXBQ, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.181709967 (213h53m40.052793002s) [noise=true , address=145.79.227.46:4601 , piece_id=AFT773CWP62ANAIFPUTDYQ35GSF6K2NFEAO7X4VKIT62CQD3NJEQ, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.183204752 (213h53m40.054287787s) [noise=true , address=145.79.227.28:4725 , piece_id=GNFCQWHHQHCCHL7VNU6TYNEFJZFGR7SMXQO66UCSNORVK4XAHPGA, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.183813983 (213h53m40.054897028s) [noise=true , address=145.79.227.45:4229 , piece_id=4IHXIUBBXYN6MZYQBRBRMJZ5HYOFNCLHKOVZKFVS6M37QMRE35OA, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.186380270 (213h53m40.057463315s) [noise=true , address=145.79.227.38:4978 , piece_id=3AV4CLNBUBIXI73A3YSDKOPIMOROFBI73TVA3KNGDSBSE6KMYNEA, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.198000036 (213h53m40.069083071s) [noise=true , address=145.79.227.36:4519 , piece_id=5X7PDP7FELCD6A5566SD7YBEUN4WELNB4IDV545B27TJ6DPEIIKA, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.201922084 (213h53m40.07300513s) [noise=true , address=145.79.227.47:4267 , piece_id=FOKTCGQ7VI4GDXRJ64LDLVSALAHAL7GRFI257S6X6G2Y4IGUNW6Q, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.217629133 (213h53m40.088712178s) [noise=true , address=145.79.227.34:4905 , piece_id=GHVXBDDAG64LE2A5RRGK3POPOL3QSQGGI7UH3NIHEZTEG7A7GODA, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.217662106 (213h53m40.088745151s) [seg_pos={0 0}, upload=187397]: Segment reached optimal threshold of 13 pieces. Dec 14 13:22:40.218130388 (213h53m40.089213423s) [noise=true , address=145.79.227.44:4193 , piece_id=UMQOWDMKE3PCVS6DQUPMMRR3IYW7EUY3ABOGSFVJ3H7KGFRUZ7HA, seg_pos={0 0}, upload=187397]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8): protocol: expected piece hash; context canceled; EOF Dec 14 13:22:40.218148793 (213h53m40.089231828s) [noise=true , address=145.79.227.44:4193 , piece_id=UMQOWDMKE3PCVS6DQUPMMRR3IYW7EUY3ABOGSFVJ3H7KGFRUZ7HA, seg_pos={0 0}, upload=187397]: Long tail cancellation due to optimal threshold Dec 14 13:22:40.218265846 (213h53m40.089348891s) [noise=true , address=145.79.227.35:4481 , piece_id=LTU57UBRXVWPXJKKRNWYSIQHL7YYPNDMI7R5GARIP55AO7KS4TXQ, seg_pos={0 0}, upload=187397]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bRK8WWmAnWUwzCfHu8JtZgrruYSJCvhnsYTKcaLccJcv5RQjp): protocol: expected piece hash; context canceled; EOF Dec 14 13:22:40.218275184 (213h53m40.089358239s) [noise=true , address=145.79.227.35:4481 , piece_id=LTU57UBRXVWPXJKKRNWYSIQHL7YYPNDMI7R5GARIP55AO7KS4TXQ, seg_pos={0 0}, upload=187397]: Long tail cancellation due to optimal threshold Dec 14 13:22:40.218315581 (213h53m40.089398616s) [noise=true , address=145.79.227.42:4123 , piece_id=ES2R3XH3RQ2UAJTCWRLZBY3VHUZQVUA2EMCNBYBQSTR3N7NWU6EQ, seg_pos={0 0}, upload=187397]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM): protocol: expected piece hash; context canceled; EOF Dec 14 13:22:40.218330358 (213h53m40.089413393s) [noise=true , address=145.79.227.42:4123 , piece_id=ES2R3XH3RQ2UAJTCWRLZBY3VHUZQVUA2EMCNBYBQSTR3N7NWU6EQ, seg_pos={0 0}, upload=187397]: Long tail cancellation due to optimal threshold Dec 14 13:22:40.218347892 (213h53m40.089430937s) [seg_pos={0 0}, upload=187397]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 14 13:22:40.218452501 (213h53m40.089535556s) [upload=187397]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 14 13:22:40.218479914 (213h53m40.089562969s) [upload=187397]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 14 13:22:43.384827291 (213h53m43.255910336s) [upload=187397]: Done uploading segments... Dec 14 13:22:43.384857198 (213h53m43.255940233s) [upload=187397]: Upload finished. err: <nil> Duration: 9.253225306s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [....................................................X.......] duration: 5.000528223s trace: https://jaeger.dev.storj.io/trace/4ab302cedffa2e5c FAIL (2025-12-14 13:32:16 UTC) orbiter warning: context deadline exceeded Duration: 5.000555263s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [................X...................................X.......] duration: 10.229820074s trace: https://jaeger.dev.storj.io/trace/2f31d4ad7317aa95 FAIL (2025-12-14 13:32:07 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 10.229865322s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 16.844575159s trace: https://jaeger.dev.storj.io/trace/8ca15e950587d46 FAIL (2025-12-14 13:32:00 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12YKLTah3swX14kL82eNogJxd8D8Hjfq7SPjDDQAapDnXmDctP1, Piece ID: GNFCQWHHQHCCHL7VNU6TYNEFJZFGR7SMXQO66UCSNORVK4XAHPGA) context deadline exceeded; piecestore: (Node ID: 125GxLsL5UtXkuJ1LFvF42N9iZkKtRK3rq63xdR25JQbtwoT1L5, Piece ID: AFT773CWP62ANAIFPUTDYQ35GSF6K2NFEAO7X4VKIT62CQD3NJEQ) context deadline exceeded ======LOGS====== Dec 14 13:22:35.112522281 (213h53m34.983605316s) [upload=187397]: Starting upload Dec 14 13:22:35.112587294 (213h53m34.983670329s) [upload=187397]: Done starting upload Dec 14 13:22:35.112673899 (213h53m34.983756934s) [upload=187397]: Uploading segments... Dec 14 13:22:35.112767347 (213h53m34.983850393s) [upload=187397]: Got next segment. Inline: false Dec 14 13:22:35.112800511 (213h53m34.983883556s) [upload=187397]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 14 13:22:35.112805821 (213h53m34.983888856s) [upload=187397]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 14 13:22:37.194903810 (213h53m37.065986845s) [seg_pos={0 0}, upload=187397]: Begin upload segment... Dec 14 13:22:37.195008499 (213h53m37.066091534s) [noise=true , address=145.79.227.42:4123 , piece_id=ES2R3XH3RQ2UAJTCWRLZBY3VHUZQVUA2EMCNBYBQSTR3N7NWU6EQ, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.195025751 (213h53m37.066108786s) [seg_pos={0 0}, upload=187397]: Done begin upload segment. Dec 14 13:22:37.195062162 (213h53m37.066145197s) [noise=true , address=145.79.227.44:4193 , piece_id=UMQOWDMKE3PCVS6DQUPMMRR3IYW7EUY3ABOGSFVJ3H7KGFRUZ7HA, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.195125401 (213h53m37.066208446s) [noise=true , address=145.79.227.45:4229 , piece_id=4IHXIUBBXYN6MZYQBRBRMJZ5HYOFNCLHKOVZKFVS6M37QMRE35OA, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.195764098 (213h53m37.066847133s) [noise=true , address=145.79.227.35:4482 , piece_id=RAUYG5T6DQIDOV5QM2OCE4GAPYIFBHUA3ZZR3Y6KN7ZX4OPMPEZA, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.196145995 (213h53m37.06722904s) [noise=true , address=145.79.227.47:4267 , piece_id=FOKTCGQ7VI4GDXRJ64LDLVSALAHAL7GRFI257S6X6G2Y4IGUNW6Q, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.196522081 (213h53m37.067605116s) [noise=true , address=145.79.227.36:4519 , piece_id=5X7PDP7FELCD6A5566SD7YBEUN4WELNB4IDV545B27TJ6DPEIIKA, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.196919738 (213h53m37.068002783s) [noise=true , address=145.79.227.40:4050 , piece_id=OFTY43B5JLDTXZRWXEHQZLUP6YJOL75CAL36EHSOQJR3GIHZOAJQ, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.197734449 (213h53m37.068817484s) [noise=true , address=145.79.227.28:4725 , piece_id=GNFCQWHHQHCCHL7VNU6TYNEFJZFGR7SMXQO66UCSNORVK4XAHPGA, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.197786599 (213h53m37.068869634s) [noise=true , address=145.79.227.43:4158 , piece_id=HDWGVVZAZI3R6KQFIUG74SWRF3G2USEMNKVNMKZDYISPRA74AUGQ, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.198256694 (213h53m37.069339739s) [noise=true , address=145.79.227.35:4481 , piece_id=LTU57UBRXVWPXJKKRNWYSIQHL7YYPNDMI7R5GARIP55AO7KS4TXQ, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.198675992 (213h53m37.069759027s) [noise=true , address=145.79.227.51:4411 , piece_id=V7ISMKJUJ6EDCLYOJIZBCBUUNDIWADZPZUUY7XH2F6BHVZILRXBQ, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.199111532 (213h53m37.070194567s) [noise=true , address=145.79.227.26:4655 , piece_id=NSYNQURKGVD7VTPMXJGF5RDIQKNUBLVKFYUE5CLPQ6IMIEJZQ4SQ, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.199658874 (213h53m37.07074191s) [noise=true , address=145.79.227.34:4905 , piece_id=GHVXBDDAG64LE2A5RRGK3POPOL3QSQGGI7UH3NIHEZTEG7A7GODA, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.200449229 (213h53m37.071532274s) [noise=true , address=145.79.227.38:4978 , piece_id=3AV4CLNBUBIXI73A3YSDKOPIMOROFBI73TVA3KNGDSBSE6KMYNEA, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.200877725 (213h53m37.07196077s) [noise=true , address=145.79.227.46:4601 , piece_id=AFT773CWP62ANAIFPUTDYQ35GSF6K2NFEAO7X4VKIT62CQD3NJEQ, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.207413279 (213h53m37.078496314s) [noise=true , address=145.79.227.31:4797 , piece_id=KRMTNOTWOWCIERSEUWBWZOIPDDUWYZKUBX74ZAIWV737CIKFVRLA, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:39.731884038 (213h53m39.602967073s) [upload=187397]: Next returned nil segment Dec 14 13:22:39.731901430 (213h53m39.602984475s) [upload=187397]: Waiting for error group managing segments... Dec 14 13:22:40.154962424 (213h53m40.026045469s) [noise=true , address=145.79.227.26:4655 , piece_id=NSYNQURKGVD7VTPMXJGF5RDIQKNUBLVKFYUE5CLPQ6IMIEJZQ4SQ, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.159418800 (213h53m40.030501835s) [noise=true , address=145.79.227.40:4050 , piece_id=OFTY43B5JLDTXZRWXEHQZLUP6YJOL75CAL36EHSOQJR3GIHZOAJQ, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.175663783 (213h53m40.046746829s) [noise=true , address=145.79.227.43:4158 , piece_id=HDWGVVZAZI3R6KQFIUG74SWRF3G2USEMNKVNMKZDYISPRA74AUGQ, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.175938396 (213h53m40.047021432s) [noise=true , address=145.79.227.35:4482 , piece_id=RAUYG5T6DQIDOV5QM2OCE4GAPYIFBHUA3ZZR3Y6KN7ZX4OPMPEZA, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.177599980 (213h53m40.048683025s) [noise=true , address=145.79.227.31:4797 , piece_id=KRMTNOTWOWCIERSEUWBWZOIPDDUWYZKUBX74ZAIWV737CIKFVRLA, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.181351754 (213h53m40.052434789s) [noise=true , address=145.79.227.51:4411 , piece_id=V7ISMKJUJ6EDCLYOJIZBCBUUNDIWADZPZUUY7XH2F6BHVZILRXBQ, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.181709967 (213h53m40.052793002s) [noise=true , address=145.79.227.46:4601 , piece_id=AFT773CWP62ANAIFPUTDYQ35GSF6K2NFEAO7X4VKIT62CQD3NJEQ, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.183204752 (213h53m40.054287787s) [noise=true , address=145.79.227.28:4725 , piece_id=GNFCQWHHQHCCHL7VNU6TYNEFJZFGR7SMXQO66UCSNORVK4XAHPGA, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.183813983 (213h53m40.054897028s) [noise=true , address=145.79.227.45:4229 , piece_id=4IHXIUBBXYN6MZYQBRBRMJZ5HYOFNCLHKOVZKFVS6M37QMRE35OA, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.186380270 (213h53m40.057463315s) [noise=true , address=145.79.227.38:4978 , piece_id=3AV4CLNBUBIXI73A3YSDKOPIMOROFBI73TVA3KNGDSBSE6KMYNEA, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.198000036 (213h53m40.069083071s) [noise=true , address=145.79.227.36:4519 , piece_id=5X7PDP7FELCD6A5566SD7YBEUN4WELNB4IDV545B27TJ6DPEIIKA, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.201922084 (213h53m40.07300513s) [noise=true , address=145.79.227.47:4267 , piece_id=FOKTCGQ7VI4GDXRJ64LDLVSALAHAL7GRFI257S6X6G2Y4IGUNW6Q, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.217629133 (213h53m40.088712178s) [noise=true , address=145.79.227.34:4905 , piece_id=GHVXBDDAG64LE2A5RRGK3POPOL3QSQGGI7UH3NIHEZTEG7A7GODA, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.217662106 (213h53m40.088745151s) [seg_pos={0 0}, upload=187397]: Segment reached optimal threshold of 13 pieces. Dec 14 13:22:40.218130388 (213h53m40.089213423s) [noise=true , address=145.79.227.44:4193 , piece_id=UMQOWDMKE3PCVS6DQUPMMRR3IYW7EUY3ABOGSFVJ3H7KGFRUZ7HA, seg_pos={0 0}, upload=187397]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8): protocol: expected piece hash; context canceled; EOF Dec 14 13:22:40.218148793 (213h53m40.089231828s) [noise=true , address=145.79.227.44:4193 , piece_id=UMQOWDMKE3PCVS6DQUPMMRR3IYW7EUY3ABOGSFVJ3H7KGFRUZ7HA, seg_pos={0 0}, upload=187397]: Long tail cancellation due to optimal threshold Dec 14 13:22:40.218265846 (213h53m40.089348891s) [noise=true , address=145.79.227.35:4481 , piece_id=LTU57UBRXVWPXJKKRNWYSIQHL7YYPNDMI7R5GARIP55AO7KS4TXQ, seg_pos={0 0}, upload=187397]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bRK8WWmAnWUwzCfHu8JtZgrruYSJCvhnsYTKcaLccJcv5RQjp): protocol: expected piece hash; context canceled; EOF Dec 14 13:22:40.218275184 (213h53m40.089358239s) [noise=true , address=145.79.227.35:4481 , piece_id=LTU57UBRXVWPXJKKRNWYSIQHL7YYPNDMI7R5GARIP55AO7KS4TXQ, seg_pos={0 0}, upload=187397]: Long tail cancellation due to optimal threshold Dec 14 13:22:40.218315581 (213h53m40.089398616s) [noise=true , address=145.79.227.42:4123 , piece_id=ES2R3XH3RQ2UAJTCWRLZBY3VHUZQVUA2EMCNBYBQSTR3N7NWU6EQ, seg_pos={0 0}, upload=187397]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM): protocol: expected piece hash; context canceled; EOF Dec 14 13:22:40.218330358 (213h53m40.089413393s) [noise=true , address=145.79.227.42:4123 , piece_id=ES2R3XH3RQ2UAJTCWRLZBY3VHUZQVUA2EMCNBYBQSTR3N7NWU6EQ, seg_pos={0 0}, upload=187397]: Long tail cancellation due to optimal threshold Dec 14 13:22:40.218347892 (213h53m40.089430937s) [seg_pos={0 0}, upload=187397]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 14 13:22:40.218452501 (213h53m40.089535556s) [upload=187397]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 14 13:22:40.218479914 (213h53m40.089562969s) [upload=187397]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 14 13:22:43.384827291 (213h53m43.255910336s) [upload=187397]: Done uploading segments... Dec 14 13:22:43.384857198 (213h53m43.255940233s) [upload=187397]: Upload finished. err: <nil> Duration: 16.844604935s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [X................X...................................X......] duration: 24.133733255s trace: https://jaeger.dev.storj.io/trace/2f31d4ad7317aa95 FAIL (2025-12-14 13:31:57 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 24.133747312s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.X................X...................................X.....] duration: 24.857929475s trace: https://jaeger.dev.storj.io/trace/2f31d4ad7317aa95 FAIL (2025-12-14 13:31:32 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 24.857958931s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.....................................................X......] duration: 5.001457891s trace: https://jaeger.dev.storj.io/trace/4ab302cedffa2e5c FAIL (2025-12-14 13:31:30 UTC) orbiter warning: context deadline exceeded Duration: 5.001482387s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 8.454318679s trace: https://jaeger.dev.storj.io/trace/8ca15e950587d46 FAIL (2025-12-14 13:31:21 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12YKLTah3swX14kL82eNogJxd8D8Hjfq7SPjDDQAapDnXmDctP1, Piece ID: GNFCQWHHQHCCHL7VNU6TYNEFJZFGR7SMXQO66UCSNORVK4XAHPGA) context deadline exceeded; piecestore: (Node ID: 125GxLsL5UtXkuJ1LFvF42N9iZkKtRK3rq63xdR25JQbtwoT1L5, Piece ID: AFT773CWP62ANAIFPUTDYQ35GSF6K2NFEAO7X4VKIT62CQD3NJEQ) context deadline exceeded ======LOGS====== Dec 14 13:22:35.112522281 (213h53m34.983605316s) [upload=187397]: Starting upload Dec 14 13:22:35.112587294 (213h53m34.983670329s) [upload=187397]: Done starting upload Dec 14 13:22:35.112673899 (213h53m34.983756934s) [upload=187397]: Uploading segments... Dec 14 13:22:35.112767347 (213h53m34.983850393s) [upload=187397]: Got next segment. Inline: false Dec 14 13:22:35.112800511 (213h53m34.983883556s) [upload=187397]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 14 13:22:35.112805821 (213h53m34.983888856s) [upload=187397]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 14 13:22:37.194903810 (213h53m37.065986845s) [seg_pos={0 0}, upload=187397]: Begin upload segment... Dec 14 13:22:37.195008499 (213h53m37.066091534s) [noise=true , address=145.79.227.42:4123 , piece_id=ES2R3XH3RQ2UAJTCWRLZBY3VHUZQVUA2EMCNBYBQSTR3N7NWU6EQ, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.195025751 (213h53m37.066108786s) [seg_pos={0 0}, upload=187397]: Done begin upload segment. Dec 14 13:22:37.195062162 (213h53m37.066145197s) [noise=true , address=145.79.227.44:4193 , piece_id=UMQOWDMKE3PCVS6DQUPMMRR3IYW7EUY3ABOGSFVJ3H7KGFRUZ7HA, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.195125401 (213h53m37.066208446s) [noise=true , address=145.79.227.45:4229 , piece_id=4IHXIUBBXYN6MZYQBRBRMJZ5HYOFNCLHKOVZKFVS6M37QMRE35OA, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.195764098 (213h53m37.066847133s) [noise=true , address=145.79.227.35:4482 , piece_id=RAUYG5T6DQIDOV5QM2OCE4GAPYIFBHUA3ZZR3Y6KN7ZX4OPMPEZA, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.196145995 (213h53m37.06722904s) [noise=true , address=145.79.227.47:4267 , piece_id=FOKTCGQ7VI4GDXRJ64LDLVSALAHAL7GRFI257S6X6G2Y4IGUNW6Q, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.196522081 (213h53m37.067605116s) [noise=true , address=145.79.227.36:4519 , piece_id=5X7PDP7FELCD6A5566SD7YBEUN4WELNB4IDV545B27TJ6DPEIIKA, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.196919738 (213h53m37.068002783s) [noise=true , address=145.79.227.40:4050 , piece_id=OFTY43B5JLDTXZRWXEHQZLUP6YJOL75CAL36EHSOQJR3GIHZOAJQ, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.197734449 (213h53m37.068817484s) [noise=true , address=145.79.227.28:4725 , piece_id=GNFCQWHHQHCCHL7VNU6TYNEFJZFGR7SMXQO66UCSNORVK4XAHPGA, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.197786599 (213h53m37.068869634s) [noise=true , address=145.79.227.43:4158 , piece_id=HDWGVVZAZI3R6KQFIUG74SWRF3G2USEMNKVNMKZDYISPRA74AUGQ, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.198256694 (213h53m37.069339739s) [noise=true , address=145.79.227.35:4481 , piece_id=LTU57UBRXVWPXJKKRNWYSIQHL7YYPNDMI7R5GARIP55AO7KS4TXQ, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.198675992 (213h53m37.069759027s) [noise=true , address=145.79.227.51:4411 , piece_id=V7ISMKJUJ6EDCLYOJIZBCBUUNDIWADZPZUUY7XH2F6BHVZILRXBQ, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.199111532 (213h53m37.070194567s) [noise=true , address=145.79.227.26:4655 , piece_id=NSYNQURKGVD7VTPMXJGF5RDIQKNUBLVKFYUE5CLPQ6IMIEJZQ4SQ, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.199658874 (213h53m37.07074191s) [noise=true , address=145.79.227.34:4905 , piece_id=GHVXBDDAG64LE2A5RRGK3POPOL3QSQGGI7UH3NIHEZTEG7A7GODA, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.200449229 (213h53m37.071532274s) [noise=true , address=145.79.227.38:4978 , piece_id=3AV4CLNBUBIXI73A3YSDKOPIMOROFBI73TVA3KNGDSBSE6KMYNEA, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.200877725 (213h53m37.07196077s) [noise=true , address=145.79.227.46:4601 , piece_id=AFT773CWP62ANAIFPUTDYQ35GSF6K2NFEAO7X4VKIT62CQD3NJEQ, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.207413279 (213h53m37.078496314s) [noise=true , address=145.79.227.31:4797 , piece_id=KRMTNOTWOWCIERSEUWBWZOIPDDUWYZKUBX74ZAIWV737CIKFVRLA, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:39.731884038 (213h53m39.602967073s) [upload=187397]: Next returned nil segment Dec 14 13:22:39.731901430 (213h53m39.602984475s) [upload=187397]: Waiting for error group managing segments... Dec 14 13:22:40.154962424 (213h53m40.026045469s) [noise=true , address=145.79.227.26:4655 , piece_id=NSYNQURKGVD7VTPMXJGF5RDIQKNUBLVKFYUE5CLPQ6IMIEJZQ4SQ, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.159418800 (213h53m40.030501835s) [noise=true , address=145.79.227.40:4050 , piece_id=OFTY43B5JLDTXZRWXEHQZLUP6YJOL75CAL36EHSOQJR3GIHZOAJQ, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.175663783 (213h53m40.046746829s) [noise=true , address=145.79.227.43:4158 , piece_id=HDWGVVZAZI3R6KQFIUG74SWRF3G2USEMNKVNMKZDYISPRA74AUGQ, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.175938396 (213h53m40.047021432s) [noise=true , address=145.79.227.35:4482 , piece_id=RAUYG5T6DQIDOV5QM2OCE4GAPYIFBHUA3ZZR3Y6KN7ZX4OPMPEZA, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.177599980 (213h53m40.048683025s) [noise=true , address=145.79.227.31:4797 , piece_id=KRMTNOTWOWCIERSEUWBWZOIPDDUWYZKUBX74ZAIWV737CIKFVRLA, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.181351754 (213h53m40.052434789s) [noise=true , address=145.79.227.51:4411 , piece_id=V7ISMKJUJ6EDCLYOJIZBCBUUNDIWADZPZUUY7XH2F6BHVZILRXBQ, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.181709967 (213h53m40.052793002s) [noise=true , address=145.79.227.46:4601 , piece_id=AFT773CWP62ANAIFPUTDYQ35GSF6K2NFEAO7X4VKIT62CQD3NJEQ, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.183204752 (213h53m40.054287787s) [noise=true , address=145.79.227.28:4725 , piece_id=GNFCQWHHQHCCHL7VNU6TYNEFJZFGR7SMXQO66UCSNORVK4XAHPGA, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.183813983 (213h53m40.054897028s) [noise=true , address=145.79.227.45:4229 , piece_id=4IHXIUBBXYN6MZYQBRBRMJZ5HYOFNCLHKOVZKFVS6M37QMRE35OA, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.186380270 (213h53m40.057463315s) [noise=true , address=145.79.227.38:4978 , piece_id=3AV4CLNBUBIXI73A3YSDKOPIMOROFBI73TVA3KNGDSBSE6KMYNEA, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.198000036 (213h53m40.069083071s) [noise=true , address=145.79.227.36:4519 , piece_id=5X7PDP7FELCD6A5566SD7YBEUN4WELNB4IDV545B27TJ6DPEIIKA, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.201922084 (213h53m40.07300513s) [noise=true , address=145.79.227.47:4267 , piece_id=FOKTCGQ7VI4GDXRJ64LDLVSALAHAL7GRFI257S6X6G2Y4IGUNW6Q, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.217629133 (213h53m40.088712178s) [noise=true , address=145.79.227.34:4905 , piece_id=GHVXBDDAG64LE2A5RRGK3POPOL3QSQGGI7UH3NIHEZTEG7A7GODA, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.217662106 (213h53m40.088745151s) [seg_pos={0 0}, upload=187397]: Segment reached optimal threshold of 13 pieces. Dec 14 13:22:40.218130388 (213h53m40.089213423s) [noise=true , address=145.79.227.44:4193 , piece_id=UMQOWDMKE3PCVS6DQUPMMRR3IYW7EUY3ABOGSFVJ3H7KGFRUZ7HA, seg_pos={0 0}, upload=187397]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8): protocol: expected piece hash; context canceled; EOF Dec 14 13:22:40.218148793 (213h53m40.089231828s) [noise=true , address=145.79.227.44:4193 , piece_id=UMQOWDMKE3PCVS6DQUPMMRR3IYW7EUY3ABOGSFVJ3H7KGFRUZ7HA, seg_pos={0 0}, upload=187397]: Long tail cancellation due to optimal threshold Dec 14 13:22:40.218265846 (213h53m40.089348891s) [noise=true , address=145.79.227.35:4481 , piece_id=LTU57UBRXVWPXJKKRNWYSIQHL7YYPNDMI7R5GARIP55AO7KS4TXQ, seg_pos={0 0}, upload=187397]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bRK8WWmAnWUwzCfHu8JtZgrruYSJCvhnsYTKcaLccJcv5RQjp): protocol: expected piece hash; context canceled; EOF Dec 14 13:22:40.218275184 (213h53m40.089358239s) [noise=true , address=145.79.227.35:4481 , piece_id=LTU57UBRXVWPXJKKRNWYSIQHL7YYPNDMI7R5GARIP55AO7KS4TXQ, seg_pos={0 0}, upload=187397]: Long tail cancellation due to optimal threshold Dec 14 13:22:40.218315581 (213h53m40.089398616s) [noise=true , address=145.79.227.42:4123 , piece_id=ES2R3XH3RQ2UAJTCWRLZBY3VHUZQVUA2EMCNBYBQSTR3N7NWU6EQ, seg_pos={0 0}, upload=187397]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM): protocol: expected piece hash; context canceled; EOF Dec 14 13:22:40.218330358 (213h53m40.089413393s) [noise=true , address=145.79.227.42:4123 , piece_id=ES2R3XH3RQ2UAJTCWRLZBY3VHUZQVUA2EMCNBYBQSTR3N7NWU6EQ, seg_pos={0 0}, upload=187397]: Long tail cancellation due to optimal threshold Dec 14 13:22:40.218347892 (213h53m40.089430937s) [seg_pos={0 0}, upload=187397]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 14 13:22:40.218452501 (213h53m40.089535556s) [upload=187397]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 14 13:22:40.218479914 (213h53m40.089562969s) [upload=187397]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 14 13:22:43.384827291 (213h53m43.255910336s) [upload=187397]: Done uploading segments... Dec 14 13:22:43.384857198 (213h53m43.255940233s) [upload=187397]: Upload finished. err: <nil> Duration: 8.45434053s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..X................X...................................X....] duration: 14.530576519s trace: https://jaeger.dev.storj.io/trace/2f31d4ad7317aa95 FAIL (2025-12-14 13:30:57 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 14.530606527s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...X................X...................................X...] duration: 21.446697499s trace: https://jaeger.dev.storj.io/trace/2f31d4ad7317aa95 FAIL (2025-12-14 13:30:40 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 21.446729047s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 17.414882875s trace: https://jaeger.dev.storj.io/trace/8ca15e950587d46 FAIL (2025-12-14 13:30:30 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12YKLTah3swX14kL82eNogJxd8D8Hjfq7SPjDDQAapDnXmDctP1, Piece ID: GNFCQWHHQHCCHL7VNU6TYNEFJZFGR7SMXQO66UCSNORVK4XAHPGA) context deadline exceeded; piecestore: (Node ID: 125GxLsL5UtXkuJ1LFvF42N9iZkKtRK3rq63xdR25JQbtwoT1L5, Piece ID: AFT773CWP62ANAIFPUTDYQ35GSF6K2NFEAO7X4VKIT62CQD3NJEQ) context deadline exceeded ======LOGS====== Dec 14 13:22:35.112522281 (213h53m34.983605316s) [upload=187397]: Starting upload Dec 14 13:22:35.112587294 (213h53m34.983670329s) [upload=187397]: Done starting upload Dec 14 13:22:35.112673899 (213h53m34.983756934s) [upload=187397]: Uploading segments... Dec 14 13:22:35.112767347 (213h53m34.983850393s) [upload=187397]: Got next segment. Inline: false Dec 14 13:22:35.112800511 (213h53m34.983883556s) [upload=187397]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 14 13:22:35.112805821 (213h53m34.983888856s) [upload=187397]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 14 13:22:37.194903810 (213h53m37.065986845s) [seg_pos={0 0}, upload=187397]: Begin upload segment... Dec 14 13:22:37.195008499 (213h53m37.066091534s) [noise=true , address=145.79.227.42:4123 , piece_id=ES2R3XH3RQ2UAJTCWRLZBY3VHUZQVUA2EMCNBYBQSTR3N7NWU6EQ, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.195025751 (213h53m37.066108786s) [seg_pos={0 0}, upload=187397]: Done begin upload segment. Dec 14 13:22:37.195062162 (213h53m37.066145197s) [noise=true , address=145.79.227.44:4193 , piece_id=UMQOWDMKE3PCVS6DQUPMMRR3IYW7EUY3ABOGSFVJ3H7KGFRUZ7HA, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.195125401 (213h53m37.066208446s) [noise=true , address=145.79.227.45:4229 , piece_id=4IHXIUBBXYN6MZYQBRBRMJZ5HYOFNCLHKOVZKFVS6M37QMRE35OA, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.195764098 (213h53m37.066847133s) [noise=true , address=145.79.227.35:4482 , piece_id=RAUYG5T6DQIDOV5QM2OCE4GAPYIFBHUA3ZZR3Y6KN7ZX4OPMPEZA, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.196145995 (213h53m37.06722904s) [noise=true , address=145.79.227.47:4267 , piece_id=FOKTCGQ7VI4GDXRJ64LDLVSALAHAL7GRFI257S6X6G2Y4IGUNW6Q, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.196522081 (213h53m37.067605116s) [noise=true , address=145.79.227.36:4519 , piece_id=5X7PDP7FELCD6A5566SD7YBEUN4WELNB4IDV545B27TJ6DPEIIKA, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.196919738 (213h53m37.068002783s) [noise=true , address=145.79.227.40:4050 , piece_id=OFTY43B5JLDTXZRWXEHQZLUP6YJOL75CAL36EHSOQJR3GIHZOAJQ, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.197734449 (213h53m37.068817484s) [noise=true , address=145.79.227.28:4725 , piece_id=GNFCQWHHQHCCHL7VNU6TYNEFJZFGR7SMXQO66UCSNORVK4XAHPGA, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.197786599 (213h53m37.068869634s) [noise=true , address=145.79.227.43:4158 , piece_id=HDWGVVZAZI3R6KQFIUG74SWRF3G2USEMNKVNMKZDYISPRA74AUGQ, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.198256694 (213h53m37.069339739s) [noise=true , address=145.79.227.35:4481 , piece_id=LTU57UBRXVWPXJKKRNWYSIQHL7YYPNDMI7R5GARIP55AO7KS4TXQ, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.198675992 (213h53m37.069759027s) [noise=true , address=145.79.227.51:4411 , piece_id=V7ISMKJUJ6EDCLYOJIZBCBUUNDIWADZPZUUY7XH2F6BHVZILRXBQ, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.199111532 (213h53m37.070194567s) [noise=true , address=145.79.227.26:4655 , piece_id=NSYNQURKGVD7VTPMXJGF5RDIQKNUBLVKFYUE5CLPQ6IMIEJZQ4SQ, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.199658874 (213h53m37.07074191s) [noise=true , address=145.79.227.34:4905 , piece_id=GHVXBDDAG64LE2A5RRGK3POPOL3QSQGGI7UH3NIHEZTEG7A7GODA, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.200449229 (213h53m37.071532274s) [noise=true , address=145.79.227.38:4978 , piece_id=3AV4CLNBUBIXI73A3YSDKOPIMOROFBI73TVA3KNGDSBSE6KMYNEA, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.200877725 (213h53m37.07196077s) [noise=true , address=145.79.227.46:4601 , piece_id=AFT773CWP62ANAIFPUTDYQ35GSF6K2NFEAO7X4VKIT62CQD3NJEQ, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.207413279 (213h53m37.078496314s) [noise=true , address=145.79.227.31:4797 , piece_id=KRMTNOTWOWCIERSEUWBWZOIPDDUWYZKUBX74ZAIWV737CIKFVRLA, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:39.731884038 (213h53m39.602967073s) [upload=187397]: Next returned nil segment Dec 14 13:22:39.731901430 (213h53m39.602984475s) [upload=187397]: Waiting for error group managing segments... Dec 14 13:22:40.154962424 (213h53m40.026045469s) [noise=true , address=145.79.227.26:4655 , piece_id=NSYNQURKGVD7VTPMXJGF5RDIQKNUBLVKFYUE5CLPQ6IMIEJZQ4SQ, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.159418800 (213h53m40.030501835s) [noise=true , address=145.79.227.40:4050 , piece_id=OFTY43B5JLDTXZRWXEHQZLUP6YJOL75CAL36EHSOQJR3GIHZOAJQ, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.175663783 (213h53m40.046746829s) [noise=true , address=145.79.227.43:4158 , piece_id=HDWGVVZAZI3R6KQFIUG74SWRF3G2USEMNKVNMKZDYISPRA74AUGQ, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.175938396 (213h53m40.047021432s) [noise=true , address=145.79.227.35:4482 , piece_id=RAUYG5T6DQIDOV5QM2OCE4GAPYIFBHUA3ZZR3Y6KN7ZX4OPMPEZA, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.177599980 (213h53m40.048683025s) [noise=true , address=145.79.227.31:4797 , piece_id=KRMTNOTWOWCIERSEUWBWZOIPDDUWYZKUBX74ZAIWV737CIKFVRLA, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.181351754 (213h53m40.052434789s) [noise=true , address=145.79.227.51:4411 , piece_id=V7ISMKJUJ6EDCLYOJIZBCBUUNDIWADZPZUUY7XH2F6BHVZILRXBQ, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.181709967 (213h53m40.052793002s) [noise=true , address=145.79.227.46:4601 , piece_id=AFT773CWP62ANAIFPUTDYQ35GSF6K2NFEAO7X4VKIT62CQD3NJEQ, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.183204752 (213h53m40.054287787s) [noise=true , address=145.79.227.28:4725 , piece_id=GNFCQWHHQHCCHL7VNU6TYNEFJZFGR7SMXQO66UCSNORVK4XAHPGA, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.183813983 (213h53m40.054897028s) [noise=true , address=145.79.227.45:4229 , piece_id=4IHXIUBBXYN6MZYQBRBRMJZ5HYOFNCLHKOVZKFVS6M37QMRE35OA, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.186380270 (213h53m40.057463315s) [noise=true , address=145.79.227.38:4978 , piece_id=3AV4CLNBUBIXI73A3YSDKOPIMOROFBI73TVA3KNGDSBSE6KMYNEA, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.198000036 (213h53m40.069083071s) [noise=true , address=145.79.227.36:4519 , piece_id=5X7PDP7FELCD6A5566SD7YBEUN4WELNB4IDV545B27TJ6DPEIIKA, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.201922084 (213h53m40.07300513s) [noise=true , address=145.79.227.47:4267 , piece_id=FOKTCGQ7VI4GDXRJ64LDLVSALAHAL7GRFI257S6X6G2Y4IGUNW6Q, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.217629133 (213h53m40.088712178s) [noise=true , address=145.79.227.34:4905 , piece_id=GHVXBDDAG64LE2A5RRGK3POPOL3QSQGGI7UH3NIHEZTEG7A7GODA, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.217662106 (213h53m40.088745151s) [seg_pos={0 0}, upload=187397]: Segment reached optimal threshold of 13 pieces. Dec 14 13:22:40.218130388 (213h53m40.089213423s) [noise=true , address=145.79.227.44:4193 , piece_id=UMQOWDMKE3PCVS6DQUPMMRR3IYW7EUY3ABOGSFVJ3H7KGFRUZ7HA, seg_pos={0 0}, upload=187397]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8): protocol: expected piece hash; context canceled; EOF Dec 14 13:22:40.218148793 (213h53m40.089231828s) [noise=true , address=145.79.227.44:4193 , piece_id=UMQOWDMKE3PCVS6DQUPMMRR3IYW7EUY3ABOGSFVJ3H7KGFRUZ7HA, seg_pos={0 0}, upload=187397]: Long tail cancellation due to optimal threshold Dec 14 13:22:40.218265846 (213h53m40.089348891s) [noise=true , address=145.79.227.35:4481 , piece_id=LTU57UBRXVWPXJKKRNWYSIQHL7YYPNDMI7R5GARIP55AO7KS4TXQ, seg_pos={0 0}, upload=187397]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bRK8WWmAnWUwzCfHu8JtZgrruYSJCvhnsYTKcaLccJcv5RQjp): protocol: expected piece hash; context canceled; EOF Dec 14 13:22:40.218275184 (213h53m40.089358239s) [noise=true , address=145.79.227.35:4481 , piece_id=LTU57UBRXVWPXJKKRNWYSIQHL7YYPNDMI7R5GARIP55AO7KS4TXQ, seg_pos={0 0}, upload=187397]: Long tail cancellation due to optimal threshold Dec 14 13:22:40.218315581 (213h53m40.089398616s) [noise=true , address=145.79.227.42:4123 , piece_id=ES2R3XH3RQ2UAJTCWRLZBY3VHUZQVUA2EMCNBYBQSTR3N7NWU6EQ, seg_pos={0 0}, upload=187397]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM): protocol: expected piece hash; context canceled; EOF Dec 14 13:22:40.218330358 (213h53m40.089413393s) [noise=true , address=145.79.227.42:4123 , piece_id=ES2R3XH3RQ2UAJTCWRLZBY3VHUZQVUA2EMCNBYBQSTR3N7NWU6EQ, seg_pos={0 0}, upload=187397]: Long tail cancellation due to optimal threshold Dec 14 13:22:40.218347892 (213h53m40.089430937s) [seg_pos={0 0}, upload=187397]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 14 13:22:40.218452501 (213h53m40.089535556s) [upload=187397]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 14 13:22:40.218479914 (213h53m40.089562969s) [upload=187397]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 14 13:22:43.384827291 (213h53m43.255910336s) [upload=187397]: Done uploading segments... Dec 14 13:22:43.384857198 (213h53m43.255940233s) [upload=187397]: Upload finished. err: <nil> Duration: 17.414915148s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [......................................................X.....] duration: 4.706909534s trace: https://jaeger.dev.storj.io/trace/4ab302cedffa2e5c FAIL (2025-12-14 13:30:19 UTC) orbiter warning: context deadline exceeded Duration: 4.70693901s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.......................................................X....] duration: 3.7220623s trace: https://jaeger.dev.storj.io/trace/4ab302cedffa2e5c FAIL (2025-12-14 13:30:02 UTC) orbiter warning: context deadline exceeded Duration: 3.722096445s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 20.919662158s trace: https://jaeger.dev.storj.io/trace/8ca15e950587d46 FAIL (2025-12-14 13:29:40 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12YKLTah3swX14kL82eNogJxd8D8Hjfq7SPjDDQAapDnXmDctP1, Piece ID: GNFCQWHHQHCCHL7VNU6TYNEFJZFGR7SMXQO66UCSNORVK4XAHPGA) context deadline exceeded; piecestore: (Node ID: 125GxLsL5UtXkuJ1LFvF42N9iZkKtRK3rq63xdR25JQbtwoT1L5, Piece ID: AFT773CWP62ANAIFPUTDYQ35GSF6K2NFEAO7X4VKIT62CQD3NJEQ) context deadline exceeded ======LOGS====== Dec 14 13:22:35.112522281 (213h53m34.983605316s) [upload=187397]: Starting upload Dec 14 13:22:35.112587294 (213h53m34.983670329s) [upload=187397]: Done starting upload Dec 14 13:22:35.112673899 (213h53m34.983756934s) [upload=187397]: Uploading segments... Dec 14 13:22:35.112767347 (213h53m34.983850393s) [upload=187397]: Got next segment. Inline: false Dec 14 13:22:35.112800511 (213h53m34.983883556s) [upload=187397]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 14 13:22:35.112805821 (213h53m34.983888856s) [upload=187397]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 14 13:22:37.194903810 (213h53m37.065986845s) [seg_pos={0 0}, upload=187397]: Begin upload segment... Dec 14 13:22:37.195008499 (213h53m37.066091534s) [noise=true , address=145.79.227.42:4123 , piece_id=ES2R3XH3RQ2UAJTCWRLZBY3VHUZQVUA2EMCNBYBQSTR3N7NWU6EQ, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.195025751 (213h53m37.066108786s) [seg_pos={0 0}, upload=187397]: Done begin upload segment. Dec 14 13:22:37.195062162 (213h53m37.066145197s) [noise=true , address=145.79.227.44:4193 , piece_id=UMQOWDMKE3PCVS6DQUPMMRR3IYW7EUY3ABOGSFVJ3H7KGFRUZ7HA, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.195125401 (213h53m37.066208446s) [noise=true , address=145.79.227.45:4229 , piece_id=4IHXIUBBXYN6MZYQBRBRMJZ5HYOFNCLHKOVZKFVS6M37QMRE35OA, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.195764098 (213h53m37.066847133s) [noise=true , address=145.79.227.35:4482 , piece_id=RAUYG5T6DQIDOV5QM2OCE4GAPYIFBHUA3ZZR3Y6KN7ZX4OPMPEZA, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.196145995 (213h53m37.06722904s) [noise=true , address=145.79.227.47:4267 , piece_id=FOKTCGQ7VI4GDXRJ64LDLVSALAHAL7GRFI257S6X6G2Y4IGUNW6Q, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.196522081 (213h53m37.067605116s) [noise=true , address=145.79.227.36:4519 , piece_id=5X7PDP7FELCD6A5566SD7YBEUN4WELNB4IDV545B27TJ6DPEIIKA, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.196919738 (213h53m37.068002783s) [noise=true , address=145.79.227.40:4050 , piece_id=OFTY43B5JLDTXZRWXEHQZLUP6YJOL75CAL36EHSOQJR3GIHZOAJQ, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.197734449 (213h53m37.068817484s) [noise=true , address=145.79.227.28:4725 , piece_id=GNFCQWHHQHCCHL7VNU6TYNEFJZFGR7SMXQO66UCSNORVK4XAHPGA, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.197786599 (213h53m37.068869634s) [noise=true , address=145.79.227.43:4158 , piece_id=HDWGVVZAZI3R6KQFIUG74SWRF3G2USEMNKVNMKZDYISPRA74AUGQ, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.198256694 (213h53m37.069339739s) [noise=true , address=145.79.227.35:4481 , piece_id=LTU57UBRXVWPXJKKRNWYSIQHL7YYPNDMI7R5GARIP55AO7KS4TXQ, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.198675992 (213h53m37.069759027s) [noise=true , address=145.79.227.51:4411 , piece_id=V7ISMKJUJ6EDCLYOJIZBCBUUNDIWADZPZUUY7XH2F6BHVZILRXBQ, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.199111532 (213h53m37.070194567s) [noise=true , address=145.79.227.26:4655 , piece_id=NSYNQURKGVD7VTPMXJGF5RDIQKNUBLVKFYUE5CLPQ6IMIEJZQ4SQ, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.199658874 (213h53m37.07074191s) [noise=true , address=145.79.227.34:4905 , piece_id=GHVXBDDAG64LE2A5RRGK3POPOL3QSQGGI7UH3NIHEZTEG7A7GODA, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.200449229 (213h53m37.071532274s) [noise=true , address=145.79.227.38:4978 , piece_id=3AV4CLNBUBIXI73A3YSDKOPIMOROFBI73TVA3KNGDSBSE6KMYNEA, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.200877725 (213h53m37.07196077s) [noise=true , address=145.79.227.46:4601 , piece_id=AFT773CWP62ANAIFPUTDYQ35GSF6K2NFEAO7X4VKIT62CQD3NJEQ, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.207413279 (213h53m37.078496314s) [noise=true , address=145.79.227.31:4797 , piece_id=KRMTNOTWOWCIERSEUWBWZOIPDDUWYZKUBX74ZAIWV737CIKFVRLA, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:39.731884038 (213h53m39.602967073s) [upload=187397]: Next returned nil segment Dec 14 13:22:39.731901430 (213h53m39.602984475s) [upload=187397]: Waiting for error group managing segments... Dec 14 13:22:40.154962424 (213h53m40.026045469s) [noise=true , address=145.79.227.26:4655 , piece_id=NSYNQURKGVD7VTPMXJGF5RDIQKNUBLVKFYUE5CLPQ6IMIEJZQ4SQ, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.159418800 (213h53m40.030501835s) [noise=true , address=145.79.227.40:4050 , piece_id=OFTY43B5JLDTXZRWXEHQZLUP6YJOL75CAL36EHSOQJR3GIHZOAJQ, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.175663783 (213h53m40.046746829s) [noise=true , address=145.79.227.43:4158 , piece_id=HDWGVVZAZI3R6KQFIUG74SWRF3G2USEMNKVNMKZDYISPRA74AUGQ, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.175938396 (213h53m40.047021432s) [noise=true , address=145.79.227.35:4482 , piece_id=RAUYG5T6DQIDOV5QM2OCE4GAPYIFBHUA3ZZR3Y6KN7ZX4OPMPEZA, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.177599980 (213h53m40.048683025s) [noise=true , address=145.79.227.31:4797 , piece_id=KRMTNOTWOWCIERSEUWBWZOIPDDUWYZKUBX74ZAIWV737CIKFVRLA, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.181351754 (213h53m40.052434789s) [noise=true , address=145.79.227.51:4411 , piece_id=V7ISMKJUJ6EDCLYOJIZBCBUUNDIWADZPZUUY7XH2F6BHVZILRXBQ, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.181709967 (213h53m40.052793002s) [noise=true , address=145.79.227.46:4601 , piece_id=AFT773CWP62ANAIFPUTDYQ35GSF6K2NFEAO7X4VKIT62CQD3NJEQ, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.183204752 (213h53m40.054287787s) [noise=true , address=145.79.227.28:4725 , piece_id=GNFCQWHHQHCCHL7VNU6TYNEFJZFGR7SMXQO66UCSNORVK4XAHPGA, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.183813983 (213h53m40.054897028s) [noise=true , address=145.79.227.45:4229 , piece_id=4IHXIUBBXYN6MZYQBRBRMJZ5HYOFNCLHKOVZKFVS6M37QMRE35OA, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.186380270 (213h53m40.057463315s) [noise=true , address=145.79.227.38:4978 , piece_id=3AV4CLNBUBIXI73A3YSDKOPIMOROFBI73TVA3KNGDSBSE6KMYNEA, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.198000036 (213h53m40.069083071s) [noise=true , address=145.79.227.36:4519 , piece_id=5X7PDP7FELCD6A5566SD7YBEUN4WELNB4IDV545B27TJ6DPEIIKA, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.201922084 (213h53m40.07300513s) [noise=true , address=145.79.227.47:4267 , piece_id=FOKTCGQ7VI4GDXRJ64LDLVSALAHAL7GRFI257S6X6G2Y4IGUNW6Q, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.217629133 (213h53m40.088712178s) [noise=true , address=145.79.227.34:4905 , piece_id=GHVXBDDAG64LE2A5RRGK3POPOL3QSQGGI7UH3NIHEZTEG7A7GODA, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.217662106 (213h53m40.088745151s) [seg_pos={0 0}, upload=187397]: Segment reached optimal threshold of 13 pieces. Dec 14 13:22:40.218130388 (213h53m40.089213423s) [noise=true , address=145.79.227.44:4193 , piece_id=UMQOWDMKE3PCVS6DQUPMMRR3IYW7EUY3ABOGSFVJ3H7KGFRUZ7HA, seg_pos={0 0}, upload=187397]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8): protocol: expected piece hash; context canceled; EOF Dec 14 13:22:40.218148793 (213h53m40.089231828s) [noise=true , address=145.79.227.44:4193 , piece_id=UMQOWDMKE3PCVS6DQUPMMRR3IYW7EUY3ABOGSFVJ3H7KGFRUZ7HA, seg_pos={0 0}, upload=187397]: Long tail cancellation due to optimal threshold Dec 14 13:22:40.218265846 (213h53m40.089348891s) [noise=true , address=145.79.227.35:4481 , piece_id=LTU57UBRXVWPXJKKRNWYSIQHL7YYPNDMI7R5GARIP55AO7KS4TXQ, seg_pos={0 0}, upload=187397]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bRK8WWmAnWUwzCfHu8JtZgrruYSJCvhnsYTKcaLccJcv5RQjp): protocol: expected piece hash; context canceled; EOF Dec 14 13:22:40.218275184 (213h53m40.089358239s) [noise=true , address=145.79.227.35:4481 , piece_id=LTU57UBRXVWPXJKKRNWYSIQHL7YYPNDMI7R5GARIP55AO7KS4TXQ, seg_pos={0 0}, upload=187397]: Long tail cancellation due to optimal threshold Dec 14 13:22:40.218315581 (213h53m40.089398616s) [noise=true , address=145.79.227.42:4123 , piece_id=ES2R3XH3RQ2UAJTCWRLZBY3VHUZQVUA2EMCNBYBQSTR3N7NWU6EQ, seg_pos={0 0}, upload=187397]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM): protocol: expected piece hash; context canceled; EOF Dec 14 13:22:40.218330358 (213h53m40.089413393s) [noise=true , address=145.79.227.42:4123 , piece_id=ES2R3XH3RQ2UAJTCWRLZBY3VHUZQVUA2EMCNBYBQSTR3N7NWU6EQ, seg_pos={0 0}, upload=187397]: Long tail cancellation due to optimal threshold Dec 14 13:22:40.218347892 (213h53m40.089430937s) [seg_pos={0 0}, upload=187397]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 14 13:22:40.218452501 (213h53m40.089535556s) [upload=187397]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 14 13:22:40.218479914 (213h53m40.089562969s) [upload=187397]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 14 13:22:43.384827291 (213h53m43.255910336s) [upload=187397]: Done uploading segments... Dec 14 13:22:43.384857198 (213h53m43.255940233s) [upload=187397]: Upload finished. err: <nil> Duration: 20.919690812s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [....X................X...................................X..] duration: 8.644203229s trace: https://jaeger.dev.storj.io/trace/2f31d4ad7317aa95 FAIL (2025-12-14 13:29:39 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 8.64422484s edge gateway us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)