Orbiter History

checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [........................................................X...] duration: 4.109174922s trace: https://jaeger.dev.storj.io/trace/4ab302cedffa2e5c FAIL (2025-12-14 13:29:33 UTC) orbiter warning: context deadline exceeded Duration: 4.109201363s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 10.455614921s trace: https://jaeger.dev.storj.io/trace/8ca15e950587d46 FAIL (2025-12-14 13:29:09 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12YKLTah3swX14kL82eNogJxd8D8Hjfq7SPjDDQAapDnXmDctP1, Piece ID: GNFCQWHHQHCCHL7VNU6TYNEFJZFGR7SMXQO66UCSNORVK4XAHPGA) context deadline exceeded; piecestore: (Node ID: 125GxLsL5UtXkuJ1LFvF42N9iZkKtRK3rq63xdR25JQbtwoT1L5, Piece ID: AFT773CWP62ANAIFPUTDYQ35GSF6K2NFEAO7X4VKIT62CQD3NJEQ) context deadline exceeded ======LOGS====== Dec 14 13:22:35.112522281 (213h53m34.983605316s) [upload=187397]: Starting upload Dec 14 13:22:35.112587294 (213h53m34.983670329s) [upload=187397]: Done starting upload Dec 14 13:22:35.112673899 (213h53m34.983756934s) [upload=187397]: Uploading segments... Dec 14 13:22:35.112767347 (213h53m34.983850393s) [upload=187397]: Got next segment. Inline: false Dec 14 13:22:35.112800511 (213h53m34.983883556s) [upload=187397]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 14 13:22:35.112805821 (213h53m34.983888856s) [upload=187397]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 14 13:22:37.194903810 (213h53m37.065986845s) [seg_pos={0 0}, upload=187397]: Begin upload segment... Dec 14 13:22:37.195008499 (213h53m37.066091534s) [noise=true , address=145.79.227.42:4123 , piece_id=ES2R3XH3RQ2UAJTCWRLZBY3VHUZQVUA2EMCNBYBQSTR3N7NWU6EQ, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.195025751 (213h53m37.066108786s) [seg_pos={0 0}, upload=187397]: Done begin upload segment. Dec 14 13:22:37.195062162 (213h53m37.066145197s) [noise=true , address=145.79.227.44:4193 , piece_id=UMQOWDMKE3PCVS6DQUPMMRR3IYW7EUY3ABOGSFVJ3H7KGFRUZ7HA, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.195125401 (213h53m37.066208446s) [noise=true , address=145.79.227.45:4229 , piece_id=4IHXIUBBXYN6MZYQBRBRMJZ5HYOFNCLHKOVZKFVS6M37QMRE35OA, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.195764098 (213h53m37.066847133s) [noise=true , address=145.79.227.35:4482 , piece_id=RAUYG5T6DQIDOV5QM2OCE4GAPYIFBHUA3ZZR3Y6KN7ZX4OPMPEZA, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.196145995 (213h53m37.06722904s) [noise=true , address=145.79.227.47:4267 , piece_id=FOKTCGQ7VI4GDXRJ64LDLVSALAHAL7GRFI257S6X6G2Y4IGUNW6Q, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.196522081 (213h53m37.067605116s) [noise=true , address=145.79.227.36:4519 , piece_id=5X7PDP7FELCD6A5566SD7YBEUN4WELNB4IDV545B27TJ6DPEIIKA, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.196919738 (213h53m37.068002783s) [noise=true , address=145.79.227.40:4050 , piece_id=OFTY43B5JLDTXZRWXEHQZLUP6YJOL75CAL36EHSOQJR3GIHZOAJQ, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.197734449 (213h53m37.068817484s) [noise=true , address=145.79.227.28:4725 , piece_id=GNFCQWHHQHCCHL7VNU6TYNEFJZFGR7SMXQO66UCSNORVK4XAHPGA, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.197786599 (213h53m37.068869634s) [noise=true , address=145.79.227.43:4158 , piece_id=HDWGVVZAZI3R6KQFIUG74SWRF3G2USEMNKVNMKZDYISPRA74AUGQ, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.198256694 (213h53m37.069339739s) [noise=true , address=145.79.227.35:4481 , piece_id=LTU57UBRXVWPXJKKRNWYSIQHL7YYPNDMI7R5GARIP55AO7KS4TXQ, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.198675992 (213h53m37.069759027s) [noise=true , address=145.79.227.51:4411 , piece_id=V7ISMKJUJ6EDCLYOJIZBCBUUNDIWADZPZUUY7XH2F6BHVZILRXBQ, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.199111532 (213h53m37.070194567s) [noise=true , address=145.79.227.26:4655 , piece_id=NSYNQURKGVD7VTPMXJGF5RDIQKNUBLVKFYUE5CLPQ6IMIEJZQ4SQ, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.199658874 (213h53m37.07074191s) [noise=true , address=145.79.227.34:4905 , piece_id=GHVXBDDAG64LE2A5RRGK3POPOL3QSQGGI7UH3NIHEZTEG7A7GODA, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.200449229 (213h53m37.071532274s) [noise=true , address=145.79.227.38:4978 , piece_id=3AV4CLNBUBIXI73A3YSDKOPIMOROFBI73TVA3KNGDSBSE6KMYNEA, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.200877725 (213h53m37.07196077s) [noise=true , address=145.79.227.46:4601 , piece_id=AFT773CWP62ANAIFPUTDYQ35GSF6K2NFEAO7X4VKIT62CQD3NJEQ, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.207413279 (213h53m37.078496314s) [noise=true , address=145.79.227.31:4797 , piece_id=KRMTNOTWOWCIERSEUWBWZOIPDDUWYZKUBX74ZAIWV737CIKFVRLA, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:39.731884038 (213h53m39.602967073s) [upload=187397]: Next returned nil segment Dec 14 13:22:39.731901430 (213h53m39.602984475s) [upload=187397]: Waiting for error group managing segments... Dec 14 13:22:40.154962424 (213h53m40.026045469s) [noise=true , address=145.79.227.26:4655 , piece_id=NSYNQURKGVD7VTPMXJGF5RDIQKNUBLVKFYUE5CLPQ6IMIEJZQ4SQ, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.159418800 (213h53m40.030501835s) [noise=true , address=145.79.227.40:4050 , piece_id=OFTY43B5JLDTXZRWXEHQZLUP6YJOL75CAL36EHSOQJR3GIHZOAJQ, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.175663783 (213h53m40.046746829s) [noise=true , address=145.79.227.43:4158 , piece_id=HDWGVVZAZI3R6KQFIUG74SWRF3G2USEMNKVNMKZDYISPRA74AUGQ, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.175938396 (213h53m40.047021432s) [noise=true , address=145.79.227.35:4482 , piece_id=RAUYG5T6DQIDOV5QM2OCE4GAPYIFBHUA3ZZR3Y6KN7ZX4OPMPEZA, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.177599980 (213h53m40.048683025s) [noise=true , address=145.79.227.31:4797 , piece_id=KRMTNOTWOWCIERSEUWBWZOIPDDUWYZKUBX74ZAIWV737CIKFVRLA, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.181351754 (213h53m40.052434789s) [noise=true , address=145.79.227.51:4411 , piece_id=V7ISMKJUJ6EDCLYOJIZBCBUUNDIWADZPZUUY7XH2F6BHVZILRXBQ, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.181709967 (213h53m40.052793002s) [noise=true , address=145.79.227.46:4601 , piece_id=AFT773CWP62ANAIFPUTDYQ35GSF6K2NFEAO7X4VKIT62CQD3NJEQ, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.183204752 (213h53m40.054287787s) [noise=true , address=145.79.227.28:4725 , piece_id=GNFCQWHHQHCCHL7VNU6TYNEFJZFGR7SMXQO66UCSNORVK4XAHPGA, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.183813983 (213h53m40.054897028s) [noise=true , address=145.79.227.45:4229 , piece_id=4IHXIUBBXYN6MZYQBRBRMJZ5HYOFNCLHKOVZKFVS6M37QMRE35OA, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.186380270 (213h53m40.057463315s) [noise=true , address=145.79.227.38:4978 , piece_id=3AV4CLNBUBIXI73A3YSDKOPIMOROFBI73TVA3KNGDSBSE6KMYNEA, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.198000036 (213h53m40.069083071s) [noise=true , address=145.79.227.36:4519 , piece_id=5X7PDP7FELCD6A5566SD7YBEUN4WELNB4IDV545B27TJ6DPEIIKA, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.201922084 (213h53m40.07300513s) [noise=true , address=145.79.227.47:4267 , piece_id=FOKTCGQ7VI4GDXRJ64LDLVSALAHAL7GRFI257S6X6G2Y4IGUNW6Q, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.217629133 (213h53m40.088712178s) [noise=true , address=145.79.227.34:4905 , piece_id=GHVXBDDAG64LE2A5RRGK3POPOL3QSQGGI7UH3NIHEZTEG7A7GODA, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.217662106 (213h53m40.088745151s) [seg_pos={0 0}, upload=187397]: Segment reached optimal threshold of 13 pieces. Dec 14 13:22:40.218130388 (213h53m40.089213423s) [noise=true , address=145.79.227.44:4193 , piece_id=UMQOWDMKE3PCVS6DQUPMMRR3IYW7EUY3ABOGSFVJ3H7KGFRUZ7HA, seg_pos={0 0}, upload=187397]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8): protocol: expected piece hash; context canceled; EOF Dec 14 13:22:40.218148793 (213h53m40.089231828s) [noise=true , address=145.79.227.44:4193 , piece_id=UMQOWDMKE3PCVS6DQUPMMRR3IYW7EUY3ABOGSFVJ3H7KGFRUZ7HA, seg_pos={0 0}, upload=187397]: Long tail cancellation due to optimal threshold Dec 14 13:22:40.218265846 (213h53m40.089348891s) [noise=true , address=145.79.227.35:4481 , piece_id=LTU57UBRXVWPXJKKRNWYSIQHL7YYPNDMI7R5GARIP55AO7KS4TXQ, seg_pos={0 0}, upload=187397]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bRK8WWmAnWUwzCfHu8JtZgrruYSJCvhnsYTKcaLccJcv5RQjp): protocol: expected piece hash; context canceled; EOF Dec 14 13:22:40.218275184 (213h53m40.089358239s) [noise=true , address=145.79.227.35:4481 , piece_id=LTU57UBRXVWPXJKKRNWYSIQHL7YYPNDMI7R5GARIP55AO7KS4TXQ, seg_pos={0 0}, upload=187397]: Long tail cancellation due to optimal threshold Dec 14 13:22:40.218315581 (213h53m40.089398616s) [noise=true , address=145.79.227.42:4123 , piece_id=ES2R3XH3RQ2UAJTCWRLZBY3VHUZQVUA2EMCNBYBQSTR3N7NWU6EQ, seg_pos={0 0}, upload=187397]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM): protocol: expected piece hash; context canceled; EOF Dec 14 13:22:40.218330358 (213h53m40.089413393s) [noise=true , address=145.79.227.42:4123 , piece_id=ES2R3XH3RQ2UAJTCWRLZBY3VHUZQVUA2EMCNBYBQSTR3N7NWU6EQ, seg_pos={0 0}, upload=187397]: Long tail cancellation due to optimal threshold Dec 14 13:22:40.218347892 (213h53m40.089430937s) [seg_pos={0 0}, upload=187397]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 14 13:22:40.218452501 (213h53m40.089535556s) [upload=187397]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 14 13:22:40.218479914 (213h53m40.089562969s) [upload=187397]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 14 13:22:43.384827291 (213h53m43.255910336s) [upload=187397]: Done uploading segments... Dec 14 13:22:43.384857198 (213h53m43.255940233s) [upload=187397]: Upload finished. err: <nil> Duration: 10.455643715s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.....X................X...................................X.] duration: 9.513090375s trace: https://jaeger.dev.storj.io/trace/2f31d4ad7317aa95 FAIL (2025-12-14 13:29:04 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 9.513131614s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 9.259483471s trace: https://jaeger.dev.storj.io/trace/8ca15e950587d46 FAIL (2025-12-14 13:28:37 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12YKLTah3swX14kL82eNogJxd8D8Hjfq7SPjDDQAapDnXmDctP1, Piece ID: GNFCQWHHQHCCHL7VNU6TYNEFJZFGR7SMXQO66UCSNORVK4XAHPGA) context deadline exceeded; piecestore: (Node ID: 125GxLsL5UtXkuJ1LFvF42N9iZkKtRK3rq63xdR25JQbtwoT1L5, Piece ID: AFT773CWP62ANAIFPUTDYQ35GSF6K2NFEAO7X4VKIT62CQD3NJEQ) context deadline exceeded ======LOGS====== Dec 14 13:22:35.112522281 (213h53m34.983605316s) [upload=187397]: Starting upload Dec 14 13:22:35.112587294 (213h53m34.983670329s) [upload=187397]: Done starting upload Dec 14 13:22:35.112673899 (213h53m34.983756934s) [upload=187397]: Uploading segments... Dec 14 13:22:35.112767347 (213h53m34.983850393s) [upload=187397]: Got next segment. Inline: false Dec 14 13:22:35.112800511 (213h53m34.983883556s) [upload=187397]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 14 13:22:35.112805821 (213h53m34.983888856s) [upload=187397]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 14 13:22:37.194903810 (213h53m37.065986845s) [seg_pos={0 0}, upload=187397]: Begin upload segment... Dec 14 13:22:37.195008499 (213h53m37.066091534s) [noise=true , address=145.79.227.42:4123 , piece_id=ES2R3XH3RQ2UAJTCWRLZBY3VHUZQVUA2EMCNBYBQSTR3N7NWU6EQ, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.195025751 (213h53m37.066108786s) [seg_pos={0 0}, upload=187397]: Done begin upload segment. Dec 14 13:22:37.195062162 (213h53m37.066145197s) [noise=true , address=145.79.227.44:4193 , piece_id=UMQOWDMKE3PCVS6DQUPMMRR3IYW7EUY3ABOGSFVJ3H7KGFRUZ7HA, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.195125401 (213h53m37.066208446s) [noise=true , address=145.79.227.45:4229 , piece_id=4IHXIUBBXYN6MZYQBRBRMJZ5HYOFNCLHKOVZKFVS6M37QMRE35OA, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.195764098 (213h53m37.066847133s) [noise=true , address=145.79.227.35:4482 , piece_id=RAUYG5T6DQIDOV5QM2OCE4GAPYIFBHUA3ZZR3Y6KN7ZX4OPMPEZA, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.196145995 (213h53m37.06722904s) [noise=true , address=145.79.227.47:4267 , piece_id=FOKTCGQ7VI4GDXRJ64LDLVSALAHAL7GRFI257S6X6G2Y4IGUNW6Q, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.196522081 (213h53m37.067605116s) [noise=true , address=145.79.227.36:4519 , piece_id=5X7PDP7FELCD6A5566SD7YBEUN4WELNB4IDV545B27TJ6DPEIIKA, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.196919738 (213h53m37.068002783s) [noise=true , address=145.79.227.40:4050 , piece_id=OFTY43B5JLDTXZRWXEHQZLUP6YJOL75CAL36EHSOQJR3GIHZOAJQ, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.197734449 (213h53m37.068817484s) [noise=true , address=145.79.227.28:4725 , piece_id=GNFCQWHHQHCCHL7VNU6TYNEFJZFGR7SMXQO66UCSNORVK4XAHPGA, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.197786599 (213h53m37.068869634s) [noise=true , address=145.79.227.43:4158 , piece_id=HDWGVVZAZI3R6KQFIUG74SWRF3G2USEMNKVNMKZDYISPRA74AUGQ, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.198256694 (213h53m37.069339739s) [noise=true , address=145.79.227.35:4481 , piece_id=LTU57UBRXVWPXJKKRNWYSIQHL7YYPNDMI7R5GARIP55AO7KS4TXQ, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.198675992 (213h53m37.069759027s) [noise=true , address=145.79.227.51:4411 , piece_id=V7ISMKJUJ6EDCLYOJIZBCBUUNDIWADZPZUUY7XH2F6BHVZILRXBQ, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.199111532 (213h53m37.070194567s) [noise=true , address=145.79.227.26:4655 , piece_id=NSYNQURKGVD7VTPMXJGF5RDIQKNUBLVKFYUE5CLPQ6IMIEJZQ4SQ, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.199658874 (213h53m37.07074191s) [noise=true , address=145.79.227.34:4905 , piece_id=GHVXBDDAG64LE2A5RRGK3POPOL3QSQGGI7UH3NIHEZTEG7A7GODA, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.200449229 (213h53m37.071532274s) [noise=true , address=145.79.227.38:4978 , piece_id=3AV4CLNBUBIXI73A3YSDKOPIMOROFBI73TVA3KNGDSBSE6KMYNEA, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.200877725 (213h53m37.07196077s) [noise=true , address=145.79.227.46:4601 , piece_id=AFT773CWP62ANAIFPUTDYQ35GSF6K2NFEAO7X4VKIT62CQD3NJEQ, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.207413279 (213h53m37.078496314s) [noise=true , address=145.79.227.31:4797 , piece_id=KRMTNOTWOWCIERSEUWBWZOIPDDUWYZKUBX74ZAIWV737CIKFVRLA, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:39.731884038 (213h53m39.602967073s) [upload=187397]: Next returned nil segment Dec 14 13:22:39.731901430 (213h53m39.602984475s) [upload=187397]: Waiting for error group managing segments... Dec 14 13:22:40.154962424 (213h53m40.026045469s) [noise=true , address=145.79.227.26:4655 , piece_id=NSYNQURKGVD7VTPMXJGF5RDIQKNUBLVKFYUE5CLPQ6IMIEJZQ4SQ, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.159418800 (213h53m40.030501835s) [noise=true , address=145.79.227.40:4050 , piece_id=OFTY43B5JLDTXZRWXEHQZLUP6YJOL75CAL36EHSOQJR3GIHZOAJQ, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.175663783 (213h53m40.046746829s) [noise=true , address=145.79.227.43:4158 , piece_id=HDWGVVZAZI3R6KQFIUG74SWRF3G2USEMNKVNMKZDYISPRA74AUGQ, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.175938396 (213h53m40.047021432s) [noise=true , address=145.79.227.35:4482 , piece_id=RAUYG5T6DQIDOV5QM2OCE4GAPYIFBHUA3ZZR3Y6KN7ZX4OPMPEZA, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.177599980 (213h53m40.048683025s) [noise=true , address=145.79.227.31:4797 , piece_id=KRMTNOTWOWCIERSEUWBWZOIPDDUWYZKUBX74ZAIWV737CIKFVRLA, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.181351754 (213h53m40.052434789s) [noise=true , address=145.79.227.51:4411 , piece_id=V7ISMKJUJ6EDCLYOJIZBCBUUNDIWADZPZUUY7XH2F6BHVZILRXBQ, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.181709967 (213h53m40.052793002s) [noise=true , address=145.79.227.46:4601 , piece_id=AFT773CWP62ANAIFPUTDYQ35GSF6K2NFEAO7X4VKIT62CQD3NJEQ, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.183204752 (213h53m40.054287787s) [noise=true , address=145.79.227.28:4725 , piece_id=GNFCQWHHQHCCHL7VNU6TYNEFJZFGR7SMXQO66UCSNORVK4XAHPGA, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.183813983 (213h53m40.054897028s) [noise=true , address=145.79.227.45:4229 , piece_id=4IHXIUBBXYN6MZYQBRBRMJZ5HYOFNCLHKOVZKFVS6M37QMRE35OA, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.186380270 (213h53m40.057463315s) [noise=true , address=145.79.227.38:4978 , piece_id=3AV4CLNBUBIXI73A3YSDKOPIMOROFBI73TVA3KNGDSBSE6KMYNEA, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.198000036 (213h53m40.069083071s) [noise=true , address=145.79.227.36:4519 , piece_id=5X7PDP7FELCD6A5566SD7YBEUN4WELNB4IDV545B27TJ6DPEIIKA, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.201922084 (213h53m40.07300513s) [noise=true , address=145.79.227.47:4267 , piece_id=FOKTCGQ7VI4GDXRJ64LDLVSALAHAL7GRFI257S6X6G2Y4IGUNW6Q, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.217629133 (213h53m40.088712178s) [noise=true , address=145.79.227.34:4905 , piece_id=GHVXBDDAG64LE2A5RRGK3POPOL3QSQGGI7UH3NIHEZTEG7A7GODA, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.217662106 (213h53m40.088745151s) [seg_pos={0 0}, upload=187397]: Segment reached optimal threshold of 13 pieces. Dec 14 13:22:40.218130388 (213h53m40.089213423s) [noise=true , address=145.79.227.44:4193 , piece_id=UMQOWDMKE3PCVS6DQUPMMRR3IYW7EUY3ABOGSFVJ3H7KGFRUZ7HA, seg_pos={0 0}, upload=187397]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8): protocol: expected piece hash; context canceled; EOF Dec 14 13:22:40.218148793 (213h53m40.089231828s) [noise=true , address=145.79.227.44:4193 , piece_id=UMQOWDMKE3PCVS6DQUPMMRR3IYW7EUY3ABOGSFVJ3H7KGFRUZ7HA, seg_pos={0 0}, upload=187397]: Long tail cancellation due to optimal threshold Dec 14 13:22:40.218265846 (213h53m40.089348891s) [noise=true , address=145.79.227.35:4481 , piece_id=LTU57UBRXVWPXJKKRNWYSIQHL7YYPNDMI7R5GARIP55AO7KS4TXQ, seg_pos={0 0}, upload=187397]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bRK8WWmAnWUwzCfHu8JtZgrruYSJCvhnsYTKcaLccJcv5RQjp): protocol: expected piece hash; context canceled; EOF Dec 14 13:22:40.218275184 (213h53m40.089358239s) [noise=true , address=145.79.227.35:4481 , piece_id=LTU57UBRXVWPXJKKRNWYSIQHL7YYPNDMI7R5GARIP55AO7KS4TXQ, seg_pos={0 0}, upload=187397]: Long tail cancellation due to optimal threshold Dec 14 13:22:40.218315581 (213h53m40.089398616s) [noise=true , address=145.79.227.42:4123 , piece_id=ES2R3XH3RQ2UAJTCWRLZBY3VHUZQVUA2EMCNBYBQSTR3N7NWU6EQ, seg_pos={0 0}, upload=187397]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM): protocol: expected piece hash; context canceled; EOF Dec 14 13:22:40.218330358 (213h53m40.089413393s) [noise=true , address=145.79.227.42:4123 , piece_id=ES2R3XH3RQ2UAJTCWRLZBY3VHUZQVUA2EMCNBYBQSTR3N7NWU6EQ, seg_pos={0 0}, upload=187397]: Long tail cancellation due to optimal threshold Dec 14 13:22:40.218347892 (213h53m40.089430937s) [seg_pos={0 0}, upload=187397]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 14 13:22:40.218452501 (213h53m40.089535556s) [upload=187397]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 14 13:22:40.218479914 (213h53m40.089562969s) [upload=187397]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 14 13:22:43.384827291 (213h53m43.255910336s) [upload=187397]: Done uploading segments... Dec 14 13:22:43.384857198 (213h53m43.255940233s) [upload=187397]: Upload finished. err: <nil> Duration: 9.259509991s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [......X................X...................................X] duration: 30.001634611s trace: https://jaeger.dev.storj.io/trace/2f31d4ad7317aa95 FAIL (2025-12-14 13:28:27 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.001659599s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.........................................................X..] duration: 4.713098391s trace: https://jaeger.dev.storj.io/trace/4ab302cedffa2e5c FAIL (2025-12-14 13:28:20 UTC) orbiter warning: context deadline exceeded Duration: 4.713159837s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 15.924470799s trace: https://jaeger.dev.storj.io/trace/8ca15e950587d46 FAIL (2025-12-14 13:27:40 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12YKLTah3swX14kL82eNogJxd8D8Hjfq7SPjDDQAapDnXmDctP1, Piece ID: GNFCQWHHQHCCHL7VNU6TYNEFJZFGR7SMXQO66UCSNORVK4XAHPGA) context deadline exceeded; piecestore: (Node ID: 125GxLsL5UtXkuJ1LFvF42N9iZkKtRK3rq63xdR25JQbtwoT1L5, Piece ID: AFT773CWP62ANAIFPUTDYQ35GSF6K2NFEAO7X4VKIT62CQD3NJEQ) context deadline exceeded ======LOGS====== Dec 14 13:22:35.112522281 (213h53m34.983605316s) [upload=187397]: Starting upload Dec 14 13:22:35.112587294 (213h53m34.983670329s) [upload=187397]: Done starting upload Dec 14 13:22:35.112673899 (213h53m34.983756934s) [upload=187397]: Uploading segments... Dec 14 13:22:35.112767347 (213h53m34.983850393s) [upload=187397]: Got next segment. Inline: false Dec 14 13:22:35.112800511 (213h53m34.983883556s) [upload=187397]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 14 13:22:35.112805821 (213h53m34.983888856s) [upload=187397]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 14 13:22:37.194903810 (213h53m37.065986845s) [seg_pos={0 0}, upload=187397]: Begin upload segment... Dec 14 13:22:37.195008499 (213h53m37.066091534s) [noise=true , address=145.79.227.42:4123 , piece_id=ES2R3XH3RQ2UAJTCWRLZBY3VHUZQVUA2EMCNBYBQSTR3N7NWU6EQ, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.195025751 (213h53m37.066108786s) [seg_pos={0 0}, upload=187397]: Done begin upload segment. Dec 14 13:22:37.195062162 (213h53m37.066145197s) [noise=true , address=145.79.227.44:4193 , piece_id=UMQOWDMKE3PCVS6DQUPMMRR3IYW7EUY3ABOGSFVJ3H7KGFRUZ7HA, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.195125401 (213h53m37.066208446s) [noise=true , address=145.79.227.45:4229 , piece_id=4IHXIUBBXYN6MZYQBRBRMJZ5HYOFNCLHKOVZKFVS6M37QMRE35OA, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.195764098 (213h53m37.066847133s) [noise=true , address=145.79.227.35:4482 , piece_id=RAUYG5T6DQIDOV5QM2OCE4GAPYIFBHUA3ZZR3Y6KN7ZX4OPMPEZA, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.196145995 (213h53m37.06722904s) [noise=true , address=145.79.227.47:4267 , piece_id=FOKTCGQ7VI4GDXRJ64LDLVSALAHAL7GRFI257S6X6G2Y4IGUNW6Q, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.196522081 (213h53m37.067605116s) [noise=true , address=145.79.227.36:4519 , piece_id=5X7PDP7FELCD6A5566SD7YBEUN4WELNB4IDV545B27TJ6DPEIIKA, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.196919738 (213h53m37.068002783s) [noise=true , address=145.79.227.40:4050 , piece_id=OFTY43B5JLDTXZRWXEHQZLUP6YJOL75CAL36EHSOQJR3GIHZOAJQ, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.197734449 (213h53m37.068817484s) [noise=true , address=145.79.227.28:4725 , piece_id=GNFCQWHHQHCCHL7VNU6TYNEFJZFGR7SMXQO66UCSNORVK4XAHPGA, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.197786599 (213h53m37.068869634s) [noise=true , address=145.79.227.43:4158 , piece_id=HDWGVVZAZI3R6KQFIUG74SWRF3G2USEMNKVNMKZDYISPRA74AUGQ, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.198256694 (213h53m37.069339739s) [noise=true , address=145.79.227.35:4481 , piece_id=LTU57UBRXVWPXJKKRNWYSIQHL7YYPNDMI7R5GARIP55AO7KS4TXQ, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.198675992 (213h53m37.069759027s) [noise=true , address=145.79.227.51:4411 , piece_id=V7ISMKJUJ6EDCLYOJIZBCBUUNDIWADZPZUUY7XH2F6BHVZILRXBQ, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.199111532 (213h53m37.070194567s) [noise=true , address=145.79.227.26:4655 , piece_id=NSYNQURKGVD7VTPMXJGF5RDIQKNUBLVKFYUE5CLPQ6IMIEJZQ4SQ, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.199658874 (213h53m37.07074191s) [noise=true , address=145.79.227.34:4905 , piece_id=GHVXBDDAG64LE2A5RRGK3POPOL3QSQGGI7UH3NIHEZTEG7A7GODA, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.200449229 (213h53m37.071532274s) [noise=true , address=145.79.227.38:4978 , piece_id=3AV4CLNBUBIXI73A3YSDKOPIMOROFBI73TVA3KNGDSBSE6KMYNEA, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.200877725 (213h53m37.07196077s) [noise=true , address=145.79.227.46:4601 , piece_id=AFT773CWP62ANAIFPUTDYQ35GSF6K2NFEAO7X4VKIT62CQD3NJEQ, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.207413279 (213h53m37.078496314s) [noise=true , address=145.79.227.31:4797 , piece_id=KRMTNOTWOWCIERSEUWBWZOIPDDUWYZKUBX74ZAIWV737CIKFVRLA, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:39.731884038 (213h53m39.602967073s) [upload=187397]: Next returned nil segment Dec 14 13:22:39.731901430 (213h53m39.602984475s) [upload=187397]: Waiting for error group managing segments... Dec 14 13:22:40.154962424 (213h53m40.026045469s) [noise=true , address=145.79.227.26:4655 , piece_id=NSYNQURKGVD7VTPMXJGF5RDIQKNUBLVKFYUE5CLPQ6IMIEJZQ4SQ, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.159418800 (213h53m40.030501835s) [noise=true , address=145.79.227.40:4050 , piece_id=OFTY43B5JLDTXZRWXEHQZLUP6YJOL75CAL36EHSOQJR3GIHZOAJQ, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.175663783 (213h53m40.046746829s) [noise=true , address=145.79.227.43:4158 , piece_id=HDWGVVZAZI3R6KQFIUG74SWRF3G2USEMNKVNMKZDYISPRA74AUGQ, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.175938396 (213h53m40.047021432s) [noise=true , address=145.79.227.35:4482 , piece_id=RAUYG5T6DQIDOV5QM2OCE4GAPYIFBHUA3ZZR3Y6KN7ZX4OPMPEZA, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.177599980 (213h53m40.048683025s) [noise=true , address=145.79.227.31:4797 , piece_id=KRMTNOTWOWCIERSEUWBWZOIPDDUWYZKUBX74ZAIWV737CIKFVRLA, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.181351754 (213h53m40.052434789s) [noise=true , address=145.79.227.51:4411 , piece_id=V7ISMKJUJ6EDCLYOJIZBCBUUNDIWADZPZUUY7XH2F6BHVZILRXBQ, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.181709967 (213h53m40.052793002s) [noise=true , address=145.79.227.46:4601 , piece_id=AFT773CWP62ANAIFPUTDYQ35GSF6K2NFEAO7X4VKIT62CQD3NJEQ, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.183204752 (213h53m40.054287787s) [noise=true , address=145.79.227.28:4725 , piece_id=GNFCQWHHQHCCHL7VNU6TYNEFJZFGR7SMXQO66UCSNORVK4XAHPGA, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.183813983 (213h53m40.054897028s) [noise=true , address=145.79.227.45:4229 , piece_id=4IHXIUBBXYN6MZYQBRBRMJZ5HYOFNCLHKOVZKFVS6M37QMRE35OA, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.186380270 (213h53m40.057463315s) [noise=true , address=145.79.227.38:4978 , piece_id=3AV4CLNBUBIXI73A3YSDKOPIMOROFBI73TVA3KNGDSBSE6KMYNEA, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.198000036 (213h53m40.069083071s) [noise=true , address=145.79.227.36:4519 , piece_id=5X7PDP7FELCD6A5566SD7YBEUN4WELNB4IDV545B27TJ6DPEIIKA, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.201922084 (213h53m40.07300513s) [noise=true , address=145.79.227.47:4267 , piece_id=FOKTCGQ7VI4GDXRJ64LDLVSALAHAL7GRFI257S6X6G2Y4IGUNW6Q, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.217629133 (213h53m40.088712178s) [noise=true , address=145.79.227.34:4905 , piece_id=GHVXBDDAG64LE2A5RRGK3POPOL3QSQGGI7UH3NIHEZTEG7A7GODA, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.217662106 (213h53m40.088745151s) [seg_pos={0 0}, upload=187397]: Segment reached optimal threshold of 13 pieces. Dec 14 13:22:40.218130388 (213h53m40.089213423s) [noise=true , address=145.79.227.44:4193 , piece_id=UMQOWDMKE3PCVS6DQUPMMRR3IYW7EUY3ABOGSFVJ3H7KGFRUZ7HA, seg_pos={0 0}, upload=187397]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8): protocol: expected piece hash; context canceled; EOF Dec 14 13:22:40.218148793 (213h53m40.089231828s) [noise=true , address=145.79.227.44:4193 , piece_id=UMQOWDMKE3PCVS6DQUPMMRR3IYW7EUY3ABOGSFVJ3H7KGFRUZ7HA, seg_pos={0 0}, upload=187397]: Long tail cancellation due to optimal threshold Dec 14 13:22:40.218265846 (213h53m40.089348891s) [noise=true , address=145.79.227.35:4481 , piece_id=LTU57UBRXVWPXJKKRNWYSIQHL7YYPNDMI7R5GARIP55AO7KS4TXQ, seg_pos={0 0}, upload=187397]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bRK8WWmAnWUwzCfHu8JtZgrruYSJCvhnsYTKcaLccJcv5RQjp): protocol: expected piece hash; context canceled; EOF Dec 14 13:22:40.218275184 (213h53m40.089358239s) [noise=true , address=145.79.227.35:4481 , piece_id=LTU57UBRXVWPXJKKRNWYSIQHL7YYPNDMI7R5GARIP55AO7KS4TXQ, seg_pos={0 0}, upload=187397]: Long tail cancellation due to optimal threshold Dec 14 13:22:40.218315581 (213h53m40.089398616s) [noise=true , address=145.79.227.42:4123 , piece_id=ES2R3XH3RQ2UAJTCWRLZBY3VHUZQVUA2EMCNBYBQSTR3N7NWU6EQ, seg_pos={0 0}, upload=187397]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM): protocol: expected piece hash; context canceled; EOF Dec 14 13:22:40.218330358 (213h53m40.089413393s) [noise=true , address=145.79.227.42:4123 , piece_id=ES2R3XH3RQ2UAJTCWRLZBY3VHUZQVUA2EMCNBYBQSTR3N7NWU6EQ, seg_pos={0 0}, upload=187397]: Long tail cancellation due to optimal threshold Dec 14 13:22:40.218347892 (213h53m40.089430937s) [seg_pos={0 0}, upload=187397]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 14 13:22:40.218452501 (213h53m40.089535556s) [upload=187397]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 14 13:22:40.218479914 (213h53m40.089562969s) [upload=187397]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 14 13:22:43.384827291 (213h53m43.255910336s) [upload=187397]: Done uploading segments... Dec 14 13:22:43.384857198 (213h53m43.255940233s) [upload=187397]: Upload finished. err: <nil> Duration: 15.924499233s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 10.730236523s trace: https://jaeger.dev.storj.io/trace/8ca15e950587d46 FAIL (2025-12-14 13:27:16 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12YKLTah3swX14kL82eNogJxd8D8Hjfq7SPjDDQAapDnXmDctP1, Piece ID: GNFCQWHHQHCCHL7VNU6TYNEFJZFGR7SMXQO66UCSNORVK4XAHPGA) context deadline exceeded; piecestore: (Node ID: 125GxLsL5UtXkuJ1LFvF42N9iZkKtRK3rq63xdR25JQbtwoT1L5, Piece ID: AFT773CWP62ANAIFPUTDYQ35GSF6K2NFEAO7X4VKIT62CQD3NJEQ) context deadline exceeded ======LOGS====== Dec 14 13:22:35.112522281 (213h53m34.983605316s) [upload=187397]: Starting upload Dec 14 13:22:35.112587294 (213h53m34.983670329s) [upload=187397]: Done starting upload Dec 14 13:22:35.112673899 (213h53m34.983756934s) [upload=187397]: Uploading segments... Dec 14 13:22:35.112767347 (213h53m34.983850393s) [upload=187397]: Got next segment. Inline: false Dec 14 13:22:35.112800511 (213h53m34.983883556s) [upload=187397]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 14 13:22:35.112805821 (213h53m34.983888856s) [upload=187397]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 14 13:22:37.194903810 (213h53m37.065986845s) [seg_pos={0 0}, upload=187397]: Begin upload segment... Dec 14 13:22:37.195008499 (213h53m37.066091534s) [noise=true , address=145.79.227.42:4123 , piece_id=ES2R3XH3RQ2UAJTCWRLZBY3VHUZQVUA2EMCNBYBQSTR3N7NWU6EQ, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.195025751 (213h53m37.066108786s) [seg_pos={0 0}, upload=187397]: Done begin upload segment. Dec 14 13:22:37.195062162 (213h53m37.066145197s) [noise=true , address=145.79.227.44:4193 , piece_id=UMQOWDMKE3PCVS6DQUPMMRR3IYW7EUY3ABOGSFVJ3H7KGFRUZ7HA, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.195125401 (213h53m37.066208446s) [noise=true , address=145.79.227.45:4229 , piece_id=4IHXIUBBXYN6MZYQBRBRMJZ5HYOFNCLHKOVZKFVS6M37QMRE35OA, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.195764098 (213h53m37.066847133s) [noise=true , address=145.79.227.35:4482 , piece_id=RAUYG5T6DQIDOV5QM2OCE4GAPYIFBHUA3ZZR3Y6KN7ZX4OPMPEZA, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.196145995 (213h53m37.06722904s) [noise=true , address=145.79.227.47:4267 , piece_id=FOKTCGQ7VI4GDXRJ64LDLVSALAHAL7GRFI257S6X6G2Y4IGUNW6Q, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.196522081 (213h53m37.067605116s) [noise=true , address=145.79.227.36:4519 , piece_id=5X7PDP7FELCD6A5566SD7YBEUN4WELNB4IDV545B27TJ6DPEIIKA, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.196919738 (213h53m37.068002783s) [noise=true , address=145.79.227.40:4050 , piece_id=OFTY43B5JLDTXZRWXEHQZLUP6YJOL75CAL36EHSOQJR3GIHZOAJQ, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.197734449 (213h53m37.068817484s) [noise=true , address=145.79.227.28:4725 , piece_id=GNFCQWHHQHCCHL7VNU6TYNEFJZFGR7SMXQO66UCSNORVK4XAHPGA, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.197786599 (213h53m37.068869634s) [noise=true , address=145.79.227.43:4158 , piece_id=HDWGVVZAZI3R6KQFIUG74SWRF3G2USEMNKVNMKZDYISPRA74AUGQ, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.198256694 (213h53m37.069339739s) [noise=true , address=145.79.227.35:4481 , piece_id=LTU57UBRXVWPXJKKRNWYSIQHL7YYPNDMI7R5GARIP55AO7KS4TXQ, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.198675992 (213h53m37.069759027s) [noise=true , address=145.79.227.51:4411 , piece_id=V7ISMKJUJ6EDCLYOJIZBCBUUNDIWADZPZUUY7XH2F6BHVZILRXBQ, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.199111532 (213h53m37.070194567s) [noise=true , address=145.79.227.26:4655 , piece_id=NSYNQURKGVD7VTPMXJGF5RDIQKNUBLVKFYUE5CLPQ6IMIEJZQ4SQ, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.199658874 (213h53m37.07074191s) [noise=true , address=145.79.227.34:4905 , piece_id=GHVXBDDAG64LE2A5RRGK3POPOL3QSQGGI7UH3NIHEZTEG7A7GODA, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.200449229 (213h53m37.071532274s) [noise=true , address=145.79.227.38:4978 , piece_id=3AV4CLNBUBIXI73A3YSDKOPIMOROFBI73TVA3KNGDSBSE6KMYNEA, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.200877725 (213h53m37.07196077s) [noise=true , address=145.79.227.46:4601 , piece_id=AFT773CWP62ANAIFPUTDYQ35GSF6K2NFEAO7X4VKIT62CQD3NJEQ, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.207413279 (213h53m37.078496314s) [noise=true , address=145.79.227.31:4797 , piece_id=KRMTNOTWOWCIERSEUWBWZOIPDDUWYZKUBX74ZAIWV737CIKFVRLA, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:39.731884038 (213h53m39.602967073s) [upload=187397]: Next returned nil segment Dec 14 13:22:39.731901430 (213h53m39.602984475s) [upload=187397]: Waiting for error group managing segments... Dec 14 13:22:40.154962424 (213h53m40.026045469s) [noise=true , address=145.79.227.26:4655 , piece_id=NSYNQURKGVD7VTPMXJGF5RDIQKNUBLVKFYUE5CLPQ6IMIEJZQ4SQ, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.159418800 (213h53m40.030501835s) [noise=true , address=145.79.227.40:4050 , piece_id=OFTY43B5JLDTXZRWXEHQZLUP6YJOL75CAL36EHSOQJR3GIHZOAJQ, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.175663783 (213h53m40.046746829s) [noise=true , address=145.79.227.43:4158 , piece_id=HDWGVVZAZI3R6KQFIUG74SWRF3G2USEMNKVNMKZDYISPRA74AUGQ, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.175938396 (213h53m40.047021432s) [noise=true , address=145.79.227.35:4482 , piece_id=RAUYG5T6DQIDOV5QM2OCE4GAPYIFBHUA3ZZR3Y6KN7ZX4OPMPEZA, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.177599980 (213h53m40.048683025s) [noise=true , address=145.79.227.31:4797 , piece_id=KRMTNOTWOWCIERSEUWBWZOIPDDUWYZKUBX74ZAIWV737CIKFVRLA, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.181351754 (213h53m40.052434789s) [noise=true , address=145.79.227.51:4411 , piece_id=V7ISMKJUJ6EDCLYOJIZBCBUUNDIWADZPZUUY7XH2F6BHVZILRXBQ, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.181709967 (213h53m40.052793002s) [noise=true , address=145.79.227.46:4601 , piece_id=AFT773CWP62ANAIFPUTDYQ35GSF6K2NFEAO7X4VKIT62CQD3NJEQ, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.183204752 (213h53m40.054287787s) [noise=true , address=145.79.227.28:4725 , piece_id=GNFCQWHHQHCCHL7VNU6TYNEFJZFGR7SMXQO66UCSNORVK4XAHPGA, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.183813983 (213h53m40.054897028s) [noise=true , address=145.79.227.45:4229 , piece_id=4IHXIUBBXYN6MZYQBRBRMJZ5HYOFNCLHKOVZKFVS6M37QMRE35OA, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.186380270 (213h53m40.057463315s) [noise=true , address=145.79.227.38:4978 , piece_id=3AV4CLNBUBIXI73A3YSDKOPIMOROFBI73TVA3KNGDSBSE6KMYNEA, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.198000036 (213h53m40.069083071s) [noise=true , address=145.79.227.36:4519 , piece_id=5X7PDP7FELCD6A5566SD7YBEUN4WELNB4IDV545B27TJ6DPEIIKA, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.201922084 (213h53m40.07300513s) [noise=true , address=145.79.227.47:4267 , piece_id=FOKTCGQ7VI4GDXRJ64LDLVSALAHAL7GRFI257S6X6G2Y4IGUNW6Q, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.217629133 (213h53m40.088712178s) [noise=true , address=145.79.227.34:4905 , piece_id=GHVXBDDAG64LE2A5RRGK3POPOL3QSQGGI7UH3NIHEZTEG7A7GODA, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.217662106 (213h53m40.088745151s) [seg_pos={0 0}, upload=187397]: Segment reached optimal threshold of 13 pieces. Dec 14 13:22:40.218130388 (213h53m40.089213423s) [noise=true , address=145.79.227.44:4193 , piece_id=UMQOWDMKE3PCVS6DQUPMMRR3IYW7EUY3ABOGSFVJ3H7KGFRUZ7HA, seg_pos={0 0}, upload=187397]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8): protocol: expected piece hash; context canceled; EOF Dec 14 13:22:40.218148793 (213h53m40.089231828s) [noise=true , address=145.79.227.44:4193 , piece_id=UMQOWDMKE3PCVS6DQUPMMRR3IYW7EUY3ABOGSFVJ3H7KGFRUZ7HA, seg_pos={0 0}, upload=187397]: Long tail cancellation due to optimal threshold Dec 14 13:22:40.218265846 (213h53m40.089348891s) [noise=true , address=145.79.227.35:4481 , piece_id=LTU57UBRXVWPXJKKRNWYSIQHL7YYPNDMI7R5GARIP55AO7KS4TXQ, seg_pos={0 0}, upload=187397]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bRK8WWmAnWUwzCfHu8JtZgrruYSJCvhnsYTKcaLccJcv5RQjp): protocol: expected piece hash; context canceled; EOF Dec 14 13:22:40.218275184 (213h53m40.089358239s) [noise=true , address=145.79.227.35:4481 , piece_id=LTU57UBRXVWPXJKKRNWYSIQHL7YYPNDMI7R5GARIP55AO7KS4TXQ, seg_pos={0 0}, upload=187397]: Long tail cancellation due to optimal threshold Dec 14 13:22:40.218315581 (213h53m40.089398616s) [noise=true , address=145.79.227.42:4123 , piece_id=ES2R3XH3RQ2UAJTCWRLZBY3VHUZQVUA2EMCNBYBQSTR3N7NWU6EQ, seg_pos={0 0}, upload=187397]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM): protocol: expected piece hash; context canceled; EOF Dec 14 13:22:40.218330358 (213h53m40.089413393s) [noise=true , address=145.79.227.42:4123 , piece_id=ES2R3XH3RQ2UAJTCWRLZBY3VHUZQVUA2EMCNBYBQSTR3N7NWU6EQ, seg_pos={0 0}, upload=187397]: Long tail cancellation due to optimal threshold Dec 14 13:22:40.218347892 (213h53m40.089430937s) [seg_pos={0 0}, upload=187397]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 14 13:22:40.218452501 (213h53m40.089535556s) [upload=187397]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 14 13:22:40.218479914 (213h53m40.089562969s) [upload=187397]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 14 13:22:43.384827291 (213h53m43.255910336s) [upload=187397]: Done uploading segments... Dec 14 13:22:43.384857198 (213h53m43.255940233s) [upload=187397]: Upload finished. err: <nil> Duration: 10.730274355s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..........................................................X.] duration: 3.853100931s trace: https://jaeger.dev.storj.io/trace/4ab302cedffa2e5c FAIL (2025-12-14 13:27:10 UTC) orbiter warning: context deadline exceeded Duration: 3.853134315s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......X................X...................................] duration: 13.237766113s trace: https://jaeger.dev.storj.io/trace/1c9f407bc6620bcf FAIL (2025-12-14 13:27:00 UTC) orbiter warning: context deadline exceeded Duration: 13.237792173s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 24.470616019s trace: https://jaeger.dev.storj.io/trace/8ca15e950587d46 FAIL (2025-12-14 13:26:23 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12YKLTah3swX14kL82eNogJxd8D8Hjfq7SPjDDQAapDnXmDctP1, Piece ID: GNFCQWHHQHCCHL7VNU6TYNEFJZFGR7SMXQO66UCSNORVK4XAHPGA) context deadline exceeded; piecestore: (Node ID: 125GxLsL5UtXkuJ1LFvF42N9iZkKtRK3rq63xdR25JQbtwoT1L5, Piece ID: AFT773CWP62ANAIFPUTDYQ35GSF6K2NFEAO7X4VKIT62CQD3NJEQ) context deadline exceeded ======LOGS====== Dec 14 13:22:35.112522281 (213h53m34.983605316s) [upload=187397]: Starting upload Dec 14 13:22:35.112587294 (213h53m34.983670329s) [upload=187397]: Done starting upload Dec 14 13:22:35.112673899 (213h53m34.983756934s) [upload=187397]: Uploading segments... Dec 14 13:22:35.112767347 (213h53m34.983850393s) [upload=187397]: Got next segment. Inline: false Dec 14 13:22:35.112800511 (213h53m34.983883556s) [upload=187397]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 14 13:22:35.112805821 (213h53m34.983888856s) [upload=187397]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 14 13:22:37.194903810 (213h53m37.065986845s) [seg_pos={0 0}, upload=187397]: Begin upload segment... Dec 14 13:22:37.195008499 (213h53m37.066091534s) [noise=true , address=145.79.227.42:4123 , piece_id=ES2R3XH3RQ2UAJTCWRLZBY3VHUZQVUA2EMCNBYBQSTR3N7NWU6EQ, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.195025751 (213h53m37.066108786s) [seg_pos={0 0}, upload=187397]: Done begin upload segment. Dec 14 13:22:37.195062162 (213h53m37.066145197s) [noise=true , address=145.79.227.44:4193 , piece_id=UMQOWDMKE3PCVS6DQUPMMRR3IYW7EUY3ABOGSFVJ3H7KGFRUZ7HA, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.195125401 (213h53m37.066208446s) [noise=true , address=145.79.227.45:4229 , piece_id=4IHXIUBBXYN6MZYQBRBRMJZ5HYOFNCLHKOVZKFVS6M37QMRE35OA, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.195764098 (213h53m37.066847133s) [noise=true , address=145.79.227.35:4482 , piece_id=RAUYG5T6DQIDOV5QM2OCE4GAPYIFBHUA3ZZR3Y6KN7ZX4OPMPEZA, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.196145995 (213h53m37.06722904s) [noise=true , address=145.79.227.47:4267 , piece_id=FOKTCGQ7VI4GDXRJ64LDLVSALAHAL7GRFI257S6X6G2Y4IGUNW6Q, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.196522081 (213h53m37.067605116s) [noise=true , address=145.79.227.36:4519 , piece_id=5X7PDP7FELCD6A5566SD7YBEUN4WELNB4IDV545B27TJ6DPEIIKA, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.196919738 (213h53m37.068002783s) [noise=true , address=145.79.227.40:4050 , piece_id=OFTY43B5JLDTXZRWXEHQZLUP6YJOL75CAL36EHSOQJR3GIHZOAJQ, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.197734449 (213h53m37.068817484s) [noise=true , address=145.79.227.28:4725 , piece_id=GNFCQWHHQHCCHL7VNU6TYNEFJZFGR7SMXQO66UCSNORVK4XAHPGA, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.197786599 (213h53m37.068869634s) [noise=true , address=145.79.227.43:4158 , piece_id=HDWGVVZAZI3R6KQFIUG74SWRF3G2USEMNKVNMKZDYISPRA74AUGQ, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.198256694 (213h53m37.069339739s) [noise=true , address=145.79.227.35:4481 , piece_id=LTU57UBRXVWPXJKKRNWYSIQHL7YYPNDMI7R5GARIP55AO7KS4TXQ, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.198675992 (213h53m37.069759027s) [noise=true , address=145.79.227.51:4411 , piece_id=V7ISMKJUJ6EDCLYOJIZBCBUUNDIWADZPZUUY7XH2F6BHVZILRXBQ, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.199111532 (213h53m37.070194567s) [noise=true , address=145.79.227.26:4655 , piece_id=NSYNQURKGVD7VTPMXJGF5RDIQKNUBLVKFYUE5CLPQ6IMIEJZQ4SQ, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.199658874 (213h53m37.07074191s) [noise=true , address=145.79.227.34:4905 , piece_id=GHVXBDDAG64LE2A5RRGK3POPOL3QSQGGI7UH3NIHEZTEG7A7GODA, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.200449229 (213h53m37.071532274s) [noise=true , address=145.79.227.38:4978 , piece_id=3AV4CLNBUBIXI73A3YSDKOPIMOROFBI73TVA3KNGDSBSE6KMYNEA, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.200877725 (213h53m37.07196077s) [noise=true , address=145.79.227.46:4601 , piece_id=AFT773CWP62ANAIFPUTDYQ35GSF6K2NFEAO7X4VKIT62CQD3NJEQ, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.207413279 (213h53m37.078496314s) [noise=true , address=145.79.227.31:4797 , piece_id=KRMTNOTWOWCIERSEUWBWZOIPDDUWYZKUBX74ZAIWV737CIKFVRLA, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:39.731884038 (213h53m39.602967073s) [upload=187397]: Next returned nil segment Dec 14 13:22:39.731901430 (213h53m39.602984475s) [upload=187397]: Waiting for error group managing segments... Dec 14 13:22:40.154962424 (213h53m40.026045469s) [noise=true , address=145.79.227.26:4655 , piece_id=NSYNQURKGVD7VTPMXJGF5RDIQKNUBLVKFYUE5CLPQ6IMIEJZQ4SQ, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.159418800 (213h53m40.030501835s) [noise=true , address=145.79.227.40:4050 , piece_id=OFTY43B5JLDTXZRWXEHQZLUP6YJOL75CAL36EHSOQJR3GIHZOAJQ, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.175663783 (213h53m40.046746829s) [noise=true , address=145.79.227.43:4158 , piece_id=HDWGVVZAZI3R6KQFIUG74SWRF3G2USEMNKVNMKZDYISPRA74AUGQ, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.175938396 (213h53m40.047021432s) [noise=true , address=145.79.227.35:4482 , piece_id=RAUYG5T6DQIDOV5QM2OCE4GAPYIFBHUA3ZZR3Y6KN7ZX4OPMPEZA, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.177599980 (213h53m40.048683025s) [noise=true , address=145.79.227.31:4797 , piece_id=KRMTNOTWOWCIERSEUWBWZOIPDDUWYZKUBX74ZAIWV737CIKFVRLA, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.181351754 (213h53m40.052434789s) [noise=true , address=145.79.227.51:4411 , piece_id=V7ISMKJUJ6EDCLYOJIZBCBUUNDIWADZPZUUY7XH2F6BHVZILRXBQ, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.181709967 (213h53m40.052793002s) [noise=true , address=145.79.227.46:4601 , piece_id=AFT773CWP62ANAIFPUTDYQ35GSF6K2NFEAO7X4VKIT62CQD3NJEQ, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.183204752 (213h53m40.054287787s) [noise=true , address=145.79.227.28:4725 , piece_id=GNFCQWHHQHCCHL7VNU6TYNEFJZFGR7SMXQO66UCSNORVK4XAHPGA, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.183813983 (213h53m40.054897028s) [noise=true , address=145.79.227.45:4229 , piece_id=4IHXIUBBXYN6MZYQBRBRMJZ5HYOFNCLHKOVZKFVS6M37QMRE35OA, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.186380270 (213h53m40.057463315s) [noise=true , address=145.79.227.38:4978 , piece_id=3AV4CLNBUBIXI73A3YSDKOPIMOROFBI73TVA3KNGDSBSE6KMYNEA, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.198000036 (213h53m40.069083071s) [noise=true , address=145.79.227.36:4519 , piece_id=5X7PDP7FELCD6A5566SD7YBEUN4WELNB4IDV545B27TJ6DPEIIKA, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.201922084 (213h53m40.07300513s) [noise=true , address=145.79.227.47:4267 , piece_id=FOKTCGQ7VI4GDXRJ64LDLVSALAHAL7GRFI257S6X6G2Y4IGUNW6Q, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.217629133 (213h53m40.088712178s) [noise=true , address=145.79.227.34:4905 , piece_id=GHVXBDDAG64LE2A5RRGK3POPOL3QSQGGI7UH3NIHEZTEG7A7GODA, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.217662106 (213h53m40.088745151s) [seg_pos={0 0}, upload=187397]: Segment reached optimal threshold of 13 pieces. Dec 14 13:22:40.218130388 (213h53m40.089213423s) [noise=true , address=145.79.227.44:4193 , piece_id=UMQOWDMKE3PCVS6DQUPMMRR3IYW7EUY3ABOGSFVJ3H7KGFRUZ7HA, seg_pos={0 0}, upload=187397]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8): protocol: expected piece hash; context canceled; EOF Dec 14 13:22:40.218148793 (213h53m40.089231828s) [noise=true , address=145.79.227.44:4193 , piece_id=UMQOWDMKE3PCVS6DQUPMMRR3IYW7EUY3ABOGSFVJ3H7KGFRUZ7HA, seg_pos={0 0}, upload=187397]: Long tail cancellation due to optimal threshold Dec 14 13:22:40.218265846 (213h53m40.089348891s) [noise=true , address=145.79.227.35:4481 , piece_id=LTU57UBRXVWPXJKKRNWYSIQHL7YYPNDMI7R5GARIP55AO7KS4TXQ, seg_pos={0 0}, upload=187397]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bRK8WWmAnWUwzCfHu8JtZgrruYSJCvhnsYTKcaLccJcv5RQjp): protocol: expected piece hash; context canceled; EOF Dec 14 13:22:40.218275184 (213h53m40.089358239s) [noise=true , address=145.79.227.35:4481 , piece_id=LTU57UBRXVWPXJKKRNWYSIQHL7YYPNDMI7R5GARIP55AO7KS4TXQ, seg_pos={0 0}, upload=187397]: Long tail cancellation due to optimal threshold Dec 14 13:22:40.218315581 (213h53m40.089398616s) [noise=true , address=145.79.227.42:4123 , piece_id=ES2R3XH3RQ2UAJTCWRLZBY3VHUZQVUA2EMCNBYBQSTR3N7NWU6EQ, seg_pos={0 0}, upload=187397]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM): protocol: expected piece hash; context canceled; EOF Dec 14 13:22:40.218330358 (213h53m40.089413393s) [noise=true , address=145.79.227.42:4123 , piece_id=ES2R3XH3RQ2UAJTCWRLZBY3VHUZQVUA2EMCNBYBQSTR3N7NWU6EQ, seg_pos={0 0}, upload=187397]: Long tail cancellation due to optimal threshold Dec 14 13:22:40.218347892 (213h53m40.089430937s) [seg_pos={0 0}, upload=187397]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 14 13:22:40.218452501 (213h53m40.089535556s) [upload=187397]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 14 13:22:40.218479914 (213h53m40.089562969s) [upload=187397]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 14 13:22:43.384827291 (213h53m43.255910336s) [upload=187397]: Done uploading segments... Dec 14 13:22:43.384857198 (213h53m43.255940233s) [upload=187397]: Upload finished. err: <nil> Duration: 24.470645524s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...........................................................X] duration: 5.001398933s trace: https://jaeger.dev.storj.io/trace/4ab302cedffa2e5c FAIL (2025-12-14 13:26:18 UTC) orbiter warning: context deadline exceeded Duration: 5.001419792s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 16.693548493s trace: https://jaeger.dev.storj.io/trace/8ca15e950587d46 FAIL (2025-12-14 13:25:50 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12YKLTah3swX14kL82eNogJxd8D8Hjfq7SPjDDQAapDnXmDctP1, Piece ID: GNFCQWHHQHCCHL7VNU6TYNEFJZFGR7SMXQO66UCSNORVK4XAHPGA) context deadline exceeded; piecestore: (Node ID: 125GxLsL5UtXkuJ1LFvF42N9iZkKtRK3rq63xdR25JQbtwoT1L5, Piece ID: AFT773CWP62ANAIFPUTDYQ35GSF6K2NFEAO7X4VKIT62CQD3NJEQ) context deadline exceeded ======LOGS====== Dec 14 13:22:35.112522281 (213h53m34.983605316s) [upload=187397]: Starting upload Dec 14 13:22:35.112587294 (213h53m34.983670329s) [upload=187397]: Done starting upload Dec 14 13:22:35.112673899 (213h53m34.983756934s) [upload=187397]: Uploading segments... Dec 14 13:22:35.112767347 (213h53m34.983850393s) [upload=187397]: Got next segment. Inline: false Dec 14 13:22:35.112800511 (213h53m34.983883556s) [upload=187397]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 14 13:22:35.112805821 (213h53m34.983888856s) [upload=187397]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 14 13:22:37.194903810 (213h53m37.065986845s) [seg_pos={0 0}, upload=187397]: Begin upload segment... Dec 14 13:22:37.195008499 (213h53m37.066091534s) [noise=true , address=145.79.227.42:4123 , piece_id=ES2R3XH3RQ2UAJTCWRLZBY3VHUZQVUA2EMCNBYBQSTR3N7NWU6EQ, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.195025751 (213h53m37.066108786s) [seg_pos={0 0}, upload=187397]: Done begin upload segment. Dec 14 13:22:37.195062162 (213h53m37.066145197s) [noise=true , address=145.79.227.44:4193 , piece_id=UMQOWDMKE3PCVS6DQUPMMRR3IYW7EUY3ABOGSFVJ3H7KGFRUZ7HA, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.195125401 (213h53m37.066208446s) [noise=true , address=145.79.227.45:4229 , piece_id=4IHXIUBBXYN6MZYQBRBRMJZ5HYOFNCLHKOVZKFVS6M37QMRE35OA, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.195764098 (213h53m37.066847133s) [noise=true , address=145.79.227.35:4482 , piece_id=RAUYG5T6DQIDOV5QM2OCE4GAPYIFBHUA3ZZR3Y6KN7ZX4OPMPEZA, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.196145995 (213h53m37.06722904s) [noise=true , address=145.79.227.47:4267 , piece_id=FOKTCGQ7VI4GDXRJ64LDLVSALAHAL7GRFI257S6X6G2Y4IGUNW6Q, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.196522081 (213h53m37.067605116s) [noise=true , address=145.79.227.36:4519 , piece_id=5X7PDP7FELCD6A5566SD7YBEUN4WELNB4IDV545B27TJ6DPEIIKA, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.196919738 (213h53m37.068002783s) [noise=true , address=145.79.227.40:4050 , piece_id=OFTY43B5JLDTXZRWXEHQZLUP6YJOL75CAL36EHSOQJR3GIHZOAJQ, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.197734449 (213h53m37.068817484s) [noise=true , address=145.79.227.28:4725 , piece_id=GNFCQWHHQHCCHL7VNU6TYNEFJZFGR7SMXQO66UCSNORVK4XAHPGA, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.197786599 (213h53m37.068869634s) [noise=true , address=145.79.227.43:4158 , piece_id=HDWGVVZAZI3R6KQFIUG74SWRF3G2USEMNKVNMKZDYISPRA74AUGQ, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.198256694 (213h53m37.069339739s) [noise=true , address=145.79.227.35:4481 , piece_id=LTU57UBRXVWPXJKKRNWYSIQHL7YYPNDMI7R5GARIP55AO7KS4TXQ, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.198675992 (213h53m37.069759027s) [noise=true , address=145.79.227.51:4411 , piece_id=V7ISMKJUJ6EDCLYOJIZBCBUUNDIWADZPZUUY7XH2F6BHVZILRXBQ, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.199111532 (213h53m37.070194567s) [noise=true , address=145.79.227.26:4655 , piece_id=NSYNQURKGVD7VTPMXJGF5RDIQKNUBLVKFYUE5CLPQ6IMIEJZQ4SQ, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.199658874 (213h53m37.07074191s) [noise=true , address=145.79.227.34:4905 , piece_id=GHVXBDDAG64LE2A5RRGK3POPOL3QSQGGI7UH3NIHEZTEG7A7GODA, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.200449229 (213h53m37.071532274s) [noise=true , address=145.79.227.38:4978 , piece_id=3AV4CLNBUBIXI73A3YSDKOPIMOROFBI73TVA3KNGDSBSE6KMYNEA, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.200877725 (213h53m37.07196077s) [noise=true , address=145.79.227.46:4601 , piece_id=AFT773CWP62ANAIFPUTDYQ35GSF6K2NFEAO7X4VKIT62CQD3NJEQ, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.207413279 (213h53m37.078496314s) [noise=true , address=145.79.227.31:4797 , piece_id=KRMTNOTWOWCIERSEUWBWZOIPDDUWYZKUBX74ZAIWV737CIKFVRLA, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:39.731884038 (213h53m39.602967073s) [upload=187397]: Next returned nil segment Dec 14 13:22:39.731901430 (213h53m39.602984475s) [upload=187397]: Waiting for error group managing segments... Dec 14 13:22:40.154962424 (213h53m40.026045469s) [noise=true , address=145.79.227.26:4655 , piece_id=NSYNQURKGVD7VTPMXJGF5RDIQKNUBLVKFYUE5CLPQ6IMIEJZQ4SQ, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.159418800 (213h53m40.030501835s) [noise=true , address=145.79.227.40:4050 , piece_id=OFTY43B5JLDTXZRWXEHQZLUP6YJOL75CAL36EHSOQJR3GIHZOAJQ, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.175663783 (213h53m40.046746829s) [noise=true , address=145.79.227.43:4158 , piece_id=HDWGVVZAZI3R6KQFIUG74SWRF3G2USEMNKVNMKZDYISPRA74AUGQ, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.175938396 (213h53m40.047021432s) [noise=true , address=145.79.227.35:4482 , piece_id=RAUYG5T6DQIDOV5QM2OCE4GAPYIFBHUA3ZZR3Y6KN7ZX4OPMPEZA, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.177599980 (213h53m40.048683025s) [noise=true , address=145.79.227.31:4797 , piece_id=KRMTNOTWOWCIERSEUWBWZOIPDDUWYZKUBX74ZAIWV737CIKFVRLA, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.181351754 (213h53m40.052434789s) [noise=true , address=145.79.227.51:4411 , piece_id=V7ISMKJUJ6EDCLYOJIZBCBUUNDIWADZPZUUY7XH2F6BHVZILRXBQ, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.181709967 (213h53m40.052793002s) [noise=true , address=145.79.227.46:4601 , piece_id=AFT773CWP62ANAIFPUTDYQ35GSF6K2NFEAO7X4VKIT62CQD3NJEQ, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.183204752 (213h53m40.054287787s) [noise=true , address=145.79.227.28:4725 , piece_id=GNFCQWHHQHCCHL7VNU6TYNEFJZFGR7SMXQO66UCSNORVK4XAHPGA, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.183813983 (213h53m40.054897028s) [noise=true , address=145.79.227.45:4229 , piece_id=4IHXIUBBXYN6MZYQBRBRMJZ5HYOFNCLHKOVZKFVS6M37QMRE35OA, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.186380270 (213h53m40.057463315s) [noise=true , address=145.79.227.38:4978 , piece_id=3AV4CLNBUBIXI73A3YSDKOPIMOROFBI73TVA3KNGDSBSE6KMYNEA, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.198000036 (213h53m40.069083071s) [noise=true , address=145.79.227.36:4519 , piece_id=5X7PDP7FELCD6A5566SD7YBEUN4WELNB4IDV545B27TJ6DPEIIKA, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.201922084 (213h53m40.07300513s) [noise=true , address=145.79.227.47:4267 , piece_id=FOKTCGQ7VI4GDXRJ64LDLVSALAHAL7GRFI257S6X6G2Y4IGUNW6Q, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.217629133 (213h53m40.088712178s) [noise=true , address=145.79.227.34:4905 , piece_id=GHVXBDDAG64LE2A5RRGK3POPOL3QSQGGI7UH3NIHEZTEG7A7GODA, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.217662106 (213h53m40.088745151s) [seg_pos={0 0}, upload=187397]: Segment reached optimal threshold of 13 pieces. Dec 14 13:22:40.218130388 (213h53m40.089213423s) [noise=true , address=145.79.227.44:4193 , piece_id=UMQOWDMKE3PCVS6DQUPMMRR3IYW7EUY3ABOGSFVJ3H7KGFRUZ7HA, seg_pos={0 0}, upload=187397]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8): protocol: expected piece hash; context canceled; EOF Dec 14 13:22:40.218148793 (213h53m40.089231828s) [noise=true , address=145.79.227.44:4193 , piece_id=UMQOWDMKE3PCVS6DQUPMMRR3IYW7EUY3ABOGSFVJ3H7KGFRUZ7HA, seg_pos={0 0}, upload=187397]: Long tail cancellation due to optimal threshold Dec 14 13:22:40.218265846 (213h53m40.089348891s) [noise=true , address=145.79.227.35:4481 , piece_id=LTU57UBRXVWPXJKKRNWYSIQHL7YYPNDMI7R5GARIP55AO7KS4TXQ, seg_pos={0 0}, upload=187397]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bRK8WWmAnWUwzCfHu8JtZgrruYSJCvhnsYTKcaLccJcv5RQjp): protocol: expected piece hash; context canceled; EOF Dec 14 13:22:40.218275184 (213h53m40.089358239s) [noise=true , address=145.79.227.35:4481 , piece_id=LTU57UBRXVWPXJKKRNWYSIQHL7YYPNDMI7R5GARIP55AO7KS4TXQ, seg_pos={0 0}, upload=187397]: Long tail cancellation due to optimal threshold Dec 14 13:22:40.218315581 (213h53m40.089398616s) [noise=true , address=145.79.227.42:4123 , piece_id=ES2R3XH3RQ2UAJTCWRLZBY3VHUZQVUA2EMCNBYBQSTR3N7NWU6EQ, seg_pos={0 0}, upload=187397]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM): protocol: expected piece hash; context canceled; EOF Dec 14 13:22:40.218330358 (213h53m40.089413393s) [noise=true , address=145.79.227.42:4123 , piece_id=ES2R3XH3RQ2UAJTCWRLZBY3VHUZQVUA2EMCNBYBQSTR3N7NWU6EQ, seg_pos={0 0}, upload=187397]: Long tail cancellation due to optimal threshold Dec 14 13:22:40.218347892 (213h53m40.089430937s) [seg_pos={0 0}, upload=187397]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 14 13:22:40.218452501 (213h53m40.089535556s) [upload=187397]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 14 13:22:40.218479914 (213h53m40.089562969s) [upload=187397]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 14 13:22:43.384827291 (213h53m43.255910336s) [upload=187397]: Done uploading segments... Dec 14 13:22:43.384857198 (213h53m43.255940233s) [upload=187397]: Upload finished. err: <nil> Duration: 16.693576387s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [........X................X..................................] duration: 9.932950766s trace: https://jaeger.dev.storj.io/trace/1c9f407bc6620bcf FAIL (2025-12-14 13:25:45 UTC) orbiter warning: context deadline exceeded Duration: 9.932973629s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.........X................X.................................] duration: 24.994532682s trace: https://jaeger.dev.storj.io/trace/1c9f407bc6620bcf FAIL (2025-12-14 13:25:32 UTC) orbiter warning: context deadline exceeded Duration: 24.994562109s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 22.51709561s trace: https://jaeger.dev.storj.io/trace/8ca15e950587d46 FAIL (2025-12-14 13:24:44 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12YKLTah3swX14kL82eNogJxd8D8Hjfq7SPjDDQAapDnXmDctP1, Piece ID: GNFCQWHHQHCCHL7VNU6TYNEFJZFGR7SMXQO66UCSNORVK4XAHPGA) context deadline exceeded; piecestore: (Node ID: 125GxLsL5UtXkuJ1LFvF42N9iZkKtRK3rq63xdR25JQbtwoT1L5, Piece ID: AFT773CWP62ANAIFPUTDYQ35GSF6K2NFEAO7X4VKIT62CQD3NJEQ) context deadline exceeded ======LOGS====== Dec 14 13:22:35.112522281 (213h53m34.983605316s) [upload=187397]: Starting upload Dec 14 13:22:35.112587294 (213h53m34.983670329s) [upload=187397]: Done starting upload Dec 14 13:22:35.112673899 (213h53m34.983756934s) [upload=187397]: Uploading segments... Dec 14 13:22:35.112767347 (213h53m34.983850393s) [upload=187397]: Got next segment. Inline: false Dec 14 13:22:35.112800511 (213h53m34.983883556s) [upload=187397]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 14 13:22:35.112805821 (213h53m34.983888856s) [upload=187397]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 14 13:22:37.194903810 (213h53m37.065986845s) [seg_pos={0 0}, upload=187397]: Begin upload segment... Dec 14 13:22:37.195008499 (213h53m37.066091534s) [noise=true , address=145.79.227.42:4123 , piece_id=ES2R3XH3RQ2UAJTCWRLZBY3VHUZQVUA2EMCNBYBQSTR3N7NWU6EQ, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.195025751 (213h53m37.066108786s) [seg_pos={0 0}, upload=187397]: Done begin upload segment. Dec 14 13:22:37.195062162 (213h53m37.066145197s) [noise=true , address=145.79.227.44:4193 , piece_id=UMQOWDMKE3PCVS6DQUPMMRR3IYW7EUY3ABOGSFVJ3H7KGFRUZ7HA, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.195125401 (213h53m37.066208446s) [noise=true , address=145.79.227.45:4229 , piece_id=4IHXIUBBXYN6MZYQBRBRMJZ5HYOFNCLHKOVZKFVS6M37QMRE35OA, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.195764098 (213h53m37.066847133s) [noise=true , address=145.79.227.35:4482 , piece_id=RAUYG5T6DQIDOV5QM2OCE4GAPYIFBHUA3ZZR3Y6KN7ZX4OPMPEZA, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.196145995 (213h53m37.06722904s) [noise=true , address=145.79.227.47:4267 , piece_id=FOKTCGQ7VI4GDXRJ64LDLVSALAHAL7GRFI257S6X6G2Y4IGUNW6Q, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.196522081 (213h53m37.067605116s) [noise=true , address=145.79.227.36:4519 , piece_id=5X7PDP7FELCD6A5566SD7YBEUN4WELNB4IDV545B27TJ6DPEIIKA, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.196919738 (213h53m37.068002783s) [noise=true , address=145.79.227.40:4050 , piece_id=OFTY43B5JLDTXZRWXEHQZLUP6YJOL75CAL36EHSOQJR3GIHZOAJQ, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.197734449 (213h53m37.068817484s) [noise=true , address=145.79.227.28:4725 , piece_id=GNFCQWHHQHCCHL7VNU6TYNEFJZFGR7SMXQO66UCSNORVK4XAHPGA, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.197786599 (213h53m37.068869634s) [noise=true , address=145.79.227.43:4158 , piece_id=HDWGVVZAZI3R6KQFIUG74SWRF3G2USEMNKVNMKZDYISPRA74AUGQ, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.198256694 (213h53m37.069339739s) [noise=true , address=145.79.227.35:4481 , piece_id=LTU57UBRXVWPXJKKRNWYSIQHL7YYPNDMI7R5GARIP55AO7KS4TXQ, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.198675992 (213h53m37.069759027s) [noise=true , address=145.79.227.51:4411 , piece_id=V7ISMKJUJ6EDCLYOJIZBCBUUNDIWADZPZUUY7XH2F6BHVZILRXBQ, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.199111532 (213h53m37.070194567s) [noise=true , address=145.79.227.26:4655 , piece_id=NSYNQURKGVD7VTPMXJGF5RDIQKNUBLVKFYUE5CLPQ6IMIEJZQ4SQ, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.199658874 (213h53m37.07074191s) [noise=true , address=145.79.227.34:4905 , piece_id=GHVXBDDAG64LE2A5RRGK3POPOL3QSQGGI7UH3NIHEZTEG7A7GODA, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.200449229 (213h53m37.071532274s) [noise=true , address=145.79.227.38:4978 , piece_id=3AV4CLNBUBIXI73A3YSDKOPIMOROFBI73TVA3KNGDSBSE6KMYNEA, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.200877725 (213h53m37.07196077s) [noise=true , address=145.79.227.46:4601 , piece_id=AFT773CWP62ANAIFPUTDYQ35GSF6K2NFEAO7X4VKIT62CQD3NJEQ, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.207413279 (213h53m37.078496314s) [noise=true , address=145.79.227.31:4797 , piece_id=KRMTNOTWOWCIERSEUWBWZOIPDDUWYZKUBX74ZAIWV737CIKFVRLA, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:39.731884038 (213h53m39.602967073s) [upload=187397]: Next returned nil segment Dec 14 13:22:39.731901430 (213h53m39.602984475s) [upload=187397]: Waiting for error group managing segments... Dec 14 13:22:40.154962424 (213h53m40.026045469s) [noise=true , address=145.79.227.26:4655 , piece_id=NSYNQURKGVD7VTPMXJGF5RDIQKNUBLVKFYUE5CLPQ6IMIEJZQ4SQ, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.159418800 (213h53m40.030501835s) [noise=true , address=145.79.227.40:4050 , piece_id=OFTY43B5JLDTXZRWXEHQZLUP6YJOL75CAL36EHSOQJR3GIHZOAJQ, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.175663783 (213h53m40.046746829s) [noise=true , address=145.79.227.43:4158 , piece_id=HDWGVVZAZI3R6KQFIUG74SWRF3G2USEMNKVNMKZDYISPRA74AUGQ, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.175938396 (213h53m40.047021432s) [noise=true , address=145.79.227.35:4482 , piece_id=RAUYG5T6DQIDOV5QM2OCE4GAPYIFBHUA3ZZR3Y6KN7ZX4OPMPEZA, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.177599980 (213h53m40.048683025s) [noise=true , address=145.79.227.31:4797 , piece_id=KRMTNOTWOWCIERSEUWBWZOIPDDUWYZKUBX74ZAIWV737CIKFVRLA, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.181351754 (213h53m40.052434789s) [noise=true , address=145.79.227.51:4411 , piece_id=V7ISMKJUJ6EDCLYOJIZBCBUUNDIWADZPZUUY7XH2F6BHVZILRXBQ, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.181709967 (213h53m40.052793002s) [noise=true , address=145.79.227.46:4601 , piece_id=AFT773CWP62ANAIFPUTDYQ35GSF6K2NFEAO7X4VKIT62CQD3NJEQ, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.183204752 (213h53m40.054287787s) [noise=true , address=145.79.227.28:4725 , piece_id=GNFCQWHHQHCCHL7VNU6TYNEFJZFGR7SMXQO66UCSNORVK4XAHPGA, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.183813983 (213h53m40.054897028s) [noise=true , address=145.79.227.45:4229 , piece_id=4IHXIUBBXYN6MZYQBRBRMJZ5HYOFNCLHKOVZKFVS6M37QMRE35OA, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.186380270 (213h53m40.057463315s) [noise=true , address=145.79.227.38:4978 , piece_id=3AV4CLNBUBIXI73A3YSDKOPIMOROFBI73TVA3KNGDSBSE6KMYNEA, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.198000036 (213h53m40.069083071s) [noise=true , address=145.79.227.36:4519 , piece_id=5X7PDP7FELCD6A5566SD7YBEUN4WELNB4IDV545B27TJ6DPEIIKA, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.201922084 (213h53m40.07300513s) [noise=true , address=145.79.227.47:4267 , piece_id=FOKTCGQ7VI4GDXRJ64LDLVSALAHAL7GRFI257S6X6G2Y4IGUNW6Q, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.217629133 (213h53m40.088712178s) [noise=true , address=145.79.227.34:4905 , piece_id=GHVXBDDAG64LE2A5RRGK3POPOL3QSQGGI7UH3NIHEZTEG7A7GODA, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.217662106 (213h53m40.088745151s) [seg_pos={0 0}, upload=187397]: Segment reached optimal threshold of 13 pieces. Dec 14 13:22:40.218130388 (213h53m40.089213423s) [noise=true , address=145.79.227.44:4193 , piece_id=UMQOWDMKE3PCVS6DQUPMMRR3IYW7EUY3ABOGSFVJ3H7KGFRUZ7HA, seg_pos={0 0}, upload=187397]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8): protocol: expected piece hash; context canceled; EOF Dec 14 13:22:40.218148793 (213h53m40.089231828s) [noise=true , address=145.79.227.44:4193 , piece_id=UMQOWDMKE3PCVS6DQUPMMRR3IYW7EUY3ABOGSFVJ3H7KGFRUZ7HA, seg_pos={0 0}, upload=187397]: Long tail cancellation due to optimal threshold Dec 14 13:22:40.218265846 (213h53m40.089348891s) [noise=true , address=145.79.227.35:4481 , piece_id=LTU57UBRXVWPXJKKRNWYSIQHL7YYPNDMI7R5GARIP55AO7KS4TXQ, seg_pos={0 0}, upload=187397]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bRK8WWmAnWUwzCfHu8JtZgrruYSJCvhnsYTKcaLccJcv5RQjp): protocol: expected piece hash; context canceled; EOF Dec 14 13:22:40.218275184 (213h53m40.089358239s) [noise=true , address=145.79.227.35:4481 , piece_id=LTU57UBRXVWPXJKKRNWYSIQHL7YYPNDMI7R5GARIP55AO7KS4TXQ, seg_pos={0 0}, upload=187397]: Long tail cancellation due to optimal threshold Dec 14 13:22:40.218315581 (213h53m40.089398616s) [noise=true , address=145.79.227.42:4123 , piece_id=ES2R3XH3RQ2UAJTCWRLZBY3VHUZQVUA2EMCNBYBQSTR3N7NWU6EQ, seg_pos={0 0}, upload=187397]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM): protocol: expected piece hash; context canceled; EOF Dec 14 13:22:40.218330358 (213h53m40.089413393s) [noise=true , address=145.79.227.42:4123 , piece_id=ES2R3XH3RQ2UAJTCWRLZBY3VHUZQVUA2EMCNBYBQSTR3N7NWU6EQ, seg_pos={0 0}, upload=187397]: Long tail cancellation due to optimal threshold Dec 14 13:22:40.218347892 (213h53m40.089430937s) [seg_pos={0 0}, upload=187397]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 14 13:22:40.218452501 (213h53m40.089535556s) [upload=187397]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 14 13:22:40.218479914 (213h53m40.089562969s) [upload=187397]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 14 13:22:43.384827291 (213h53m43.255910336s) [upload=187397]: Done uploading segments... Dec 14 13:22:43.384857198 (213h53m43.255940233s) [upload=187397]: Upload finished. err: <nil> Duration: 22.517123211s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..........X................X................................] duration: 23.043752137s trace: https://jaeger.dev.storj.io/trace/1c9f407bc6620bcf FAIL (2025-12-14 13:24:35 UTC) orbiter warning: context deadline exceeded Duration: 23.043828914s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 12.135805481s trace: https://jaeger.dev.storj.io/trace/8ca15e950587d46 FAIL (2025-12-14 13:24:04 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12YKLTah3swX14kL82eNogJxd8D8Hjfq7SPjDDQAapDnXmDctP1, Piece ID: GNFCQWHHQHCCHL7VNU6TYNEFJZFGR7SMXQO66UCSNORVK4XAHPGA) context deadline exceeded; piecestore: (Node ID: 125GxLsL5UtXkuJ1LFvF42N9iZkKtRK3rq63xdR25JQbtwoT1L5, Piece ID: AFT773CWP62ANAIFPUTDYQ35GSF6K2NFEAO7X4VKIT62CQD3NJEQ) context deadline exceeded ======LOGS====== Dec 14 13:22:35.112522281 (213h53m34.983605316s) [upload=187397]: Starting upload Dec 14 13:22:35.112587294 (213h53m34.983670329s) [upload=187397]: Done starting upload Dec 14 13:22:35.112673899 (213h53m34.983756934s) [upload=187397]: Uploading segments... Dec 14 13:22:35.112767347 (213h53m34.983850393s) [upload=187397]: Got next segment. Inline: false Dec 14 13:22:35.112800511 (213h53m34.983883556s) [upload=187397]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 14 13:22:35.112805821 (213h53m34.983888856s) [upload=187397]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 14 13:22:37.194903810 (213h53m37.065986845s) [seg_pos={0 0}, upload=187397]: Begin upload segment... Dec 14 13:22:37.195008499 (213h53m37.066091534s) [noise=true , address=145.79.227.42:4123 , piece_id=ES2R3XH3RQ2UAJTCWRLZBY3VHUZQVUA2EMCNBYBQSTR3N7NWU6EQ, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.195025751 (213h53m37.066108786s) [seg_pos={0 0}, upload=187397]: Done begin upload segment. Dec 14 13:22:37.195062162 (213h53m37.066145197s) [noise=true , address=145.79.227.44:4193 , piece_id=UMQOWDMKE3PCVS6DQUPMMRR3IYW7EUY3ABOGSFVJ3H7KGFRUZ7HA, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.195125401 (213h53m37.066208446s) [noise=true , address=145.79.227.45:4229 , piece_id=4IHXIUBBXYN6MZYQBRBRMJZ5HYOFNCLHKOVZKFVS6M37QMRE35OA, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.195764098 (213h53m37.066847133s) [noise=true , address=145.79.227.35:4482 , piece_id=RAUYG5T6DQIDOV5QM2OCE4GAPYIFBHUA3ZZR3Y6KN7ZX4OPMPEZA, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.196145995 (213h53m37.06722904s) [noise=true , address=145.79.227.47:4267 , piece_id=FOKTCGQ7VI4GDXRJ64LDLVSALAHAL7GRFI257S6X6G2Y4IGUNW6Q, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.196522081 (213h53m37.067605116s) [noise=true , address=145.79.227.36:4519 , piece_id=5X7PDP7FELCD6A5566SD7YBEUN4WELNB4IDV545B27TJ6DPEIIKA, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.196919738 (213h53m37.068002783s) [noise=true , address=145.79.227.40:4050 , piece_id=OFTY43B5JLDTXZRWXEHQZLUP6YJOL75CAL36EHSOQJR3GIHZOAJQ, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.197734449 (213h53m37.068817484s) [noise=true , address=145.79.227.28:4725 , piece_id=GNFCQWHHQHCCHL7VNU6TYNEFJZFGR7SMXQO66UCSNORVK4XAHPGA, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.197786599 (213h53m37.068869634s) [noise=true , address=145.79.227.43:4158 , piece_id=HDWGVVZAZI3R6KQFIUG74SWRF3G2USEMNKVNMKZDYISPRA74AUGQ, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.198256694 (213h53m37.069339739s) [noise=true , address=145.79.227.35:4481 , piece_id=LTU57UBRXVWPXJKKRNWYSIQHL7YYPNDMI7R5GARIP55AO7KS4TXQ, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.198675992 (213h53m37.069759027s) [noise=true , address=145.79.227.51:4411 , piece_id=V7ISMKJUJ6EDCLYOJIZBCBUUNDIWADZPZUUY7XH2F6BHVZILRXBQ, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.199111532 (213h53m37.070194567s) [noise=true , address=145.79.227.26:4655 , piece_id=NSYNQURKGVD7VTPMXJGF5RDIQKNUBLVKFYUE5CLPQ6IMIEJZQ4SQ, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.199658874 (213h53m37.07074191s) [noise=true , address=145.79.227.34:4905 , piece_id=GHVXBDDAG64LE2A5RRGK3POPOL3QSQGGI7UH3NIHEZTEG7A7GODA, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.200449229 (213h53m37.071532274s) [noise=true , address=145.79.227.38:4978 , piece_id=3AV4CLNBUBIXI73A3YSDKOPIMOROFBI73TVA3KNGDSBSE6KMYNEA, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.200877725 (213h53m37.07196077s) [noise=true , address=145.79.227.46:4601 , piece_id=AFT773CWP62ANAIFPUTDYQ35GSF6K2NFEAO7X4VKIT62CQD3NJEQ, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.207413279 (213h53m37.078496314s) [noise=true , address=145.79.227.31:4797 , piece_id=KRMTNOTWOWCIERSEUWBWZOIPDDUWYZKUBX74ZAIWV737CIKFVRLA, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:39.731884038 (213h53m39.602967073s) [upload=187397]: Next returned nil segment Dec 14 13:22:39.731901430 (213h53m39.602984475s) [upload=187397]: Waiting for error group managing segments... Dec 14 13:22:40.154962424 (213h53m40.026045469s) [noise=true , address=145.79.227.26:4655 , piece_id=NSYNQURKGVD7VTPMXJGF5RDIQKNUBLVKFYUE5CLPQ6IMIEJZQ4SQ, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.159418800 (213h53m40.030501835s) [noise=true , address=145.79.227.40:4050 , piece_id=OFTY43B5JLDTXZRWXEHQZLUP6YJOL75CAL36EHSOQJR3GIHZOAJQ, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.175663783 (213h53m40.046746829s) [noise=true , address=145.79.227.43:4158 , piece_id=HDWGVVZAZI3R6KQFIUG74SWRF3G2USEMNKVNMKZDYISPRA74AUGQ, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.175938396 (213h53m40.047021432s) [noise=true , address=145.79.227.35:4482 , piece_id=RAUYG5T6DQIDOV5QM2OCE4GAPYIFBHUA3ZZR3Y6KN7ZX4OPMPEZA, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.177599980 (213h53m40.048683025s) [noise=true , address=145.79.227.31:4797 , piece_id=KRMTNOTWOWCIERSEUWBWZOIPDDUWYZKUBX74ZAIWV737CIKFVRLA, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.181351754 (213h53m40.052434789s) [noise=true , address=145.79.227.51:4411 , piece_id=V7ISMKJUJ6EDCLYOJIZBCBUUNDIWADZPZUUY7XH2F6BHVZILRXBQ, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.181709967 (213h53m40.052793002s) [noise=true , address=145.79.227.46:4601 , piece_id=AFT773CWP62ANAIFPUTDYQ35GSF6K2NFEAO7X4VKIT62CQD3NJEQ, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.183204752 (213h53m40.054287787s) [noise=true , address=145.79.227.28:4725 , piece_id=GNFCQWHHQHCCHL7VNU6TYNEFJZFGR7SMXQO66UCSNORVK4XAHPGA, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.183813983 (213h53m40.054897028s) [noise=true , address=145.79.227.45:4229 , piece_id=4IHXIUBBXYN6MZYQBRBRMJZ5HYOFNCLHKOVZKFVS6M37QMRE35OA, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.186380270 (213h53m40.057463315s) [noise=true , address=145.79.227.38:4978 , piece_id=3AV4CLNBUBIXI73A3YSDKOPIMOROFBI73TVA3KNGDSBSE6KMYNEA, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.198000036 (213h53m40.069083071s) [noise=true , address=145.79.227.36:4519 , piece_id=5X7PDP7FELCD6A5566SD7YBEUN4WELNB4IDV545B27TJ6DPEIIKA, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.201922084 (213h53m40.07300513s) [noise=true , address=145.79.227.47:4267 , piece_id=FOKTCGQ7VI4GDXRJ64LDLVSALAHAL7GRFI257S6X6G2Y4IGUNW6Q, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.217629133 (213h53m40.088712178s) [noise=true , address=145.79.227.34:4905 , piece_id=GHVXBDDAG64LE2A5RRGK3POPOL3QSQGGI7UH3NIHEZTEG7A7GODA, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.217662106 (213h53m40.088745151s) [seg_pos={0 0}, upload=187397]: Segment reached optimal threshold of 13 pieces. Dec 14 13:22:40.218130388 (213h53m40.089213423s) [noise=true , address=145.79.227.44:4193 , piece_id=UMQOWDMKE3PCVS6DQUPMMRR3IYW7EUY3ABOGSFVJ3H7KGFRUZ7HA, seg_pos={0 0}, upload=187397]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8): protocol: expected piece hash; context canceled; EOF Dec 14 13:22:40.218148793 (213h53m40.089231828s) [noise=true , address=145.79.227.44:4193 , piece_id=UMQOWDMKE3PCVS6DQUPMMRR3IYW7EUY3ABOGSFVJ3H7KGFRUZ7HA, seg_pos={0 0}, upload=187397]: Long tail cancellation due to optimal threshold Dec 14 13:22:40.218265846 (213h53m40.089348891s) [noise=true , address=145.79.227.35:4481 , piece_id=LTU57UBRXVWPXJKKRNWYSIQHL7YYPNDMI7R5GARIP55AO7KS4TXQ, seg_pos={0 0}, upload=187397]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bRK8WWmAnWUwzCfHu8JtZgrruYSJCvhnsYTKcaLccJcv5RQjp): protocol: expected piece hash; context canceled; EOF Dec 14 13:22:40.218275184 (213h53m40.089358239s) [noise=true , address=145.79.227.35:4481 , piece_id=LTU57UBRXVWPXJKKRNWYSIQHL7YYPNDMI7R5GARIP55AO7KS4TXQ, seg_pos={0 0}, upload=187397]: Long tail cancellation due to optimal threshold Dec 14 13:22:40.218315581 (213h53m40.089398616s) [noise=true , address=145.79.227.42:4123 , piece_id=ES2R3XH3RQ2UAJTCWRLZBY3VHUZQVUA2EMCNBYBQSTR3N7NWU6EQ, seg_pos={0 0}, upload=187397]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM): protocol: expected piece hash; context canceled; EOF Dec 14 13:22:40.218330358 (213h53m40.089413393s) [noise=true , address=145.79.227.42:4123 , piece_id=ES2R3XH3RQ2UAJTCWRLZBY3VHUZQVUA2EMCNBYBQSTR3N7NWU6EQ, seg_pos={0 0}, upload=187397]: Long tail cancellation due to optimal threshold Dec 14 13:22:40.218347892 (213h53m40.089430937s) [seg_pos={0 0}, upload=187397]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 14 13:22:40.218452501 (213h53m40.089535556s) [upload=187397]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 14 13:22:40.218479914 (213h53m40.089562969s) [upload=187397]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 14 13:22:43.384827291 (213h53m43.255910336s) [upload=187397]: Done uploading segments... Dec 14 13:22:43.384857198 (213h53m43.255940233s) [upload=187397]: Upload finished. err: <nil> Duration: 12.13583704s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...........X................X...............................] duration: 30.000736466s trace: https://jaeger.dev.storj.io/trace/1c9f407bc6620bcf FAIL (2025-12-14 13:23:58 UTC) orbiter warning: context deadline exceeded Duration: 30.000783688s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 14.673165547s trace: https://jaeger.dev.storj.io/trace/8ca15e950587d46 FAIL (2025-12-14 13:23:19 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12YKLTah3swX14kL82eNogJxd8D8Hjfq7SPjDDQAapDnXmDctP1, Piece ID: GNFCQWHHQHCCHL7VNU6TYNEFJZFGR7SMXQO66UCSNORVK4XAHPGA) context deadline exceeded; piecestore: (Node ID: 125GxLsL5UtXkuJ1LFvF42N9iZkKtRK3rq63xdR25JQbtwoT1L5, Piece ID: AFT773CWP62ANAIFPUTDYQ35GSF6K2NFEAO7X4VKIT62CQD3NJEQ) context deadline exceeded ======LOGS====== Dec 14 13:22:35.112522281 (213h53m34.983605316s) [upload=187397]: Starting upload Dec 14 13:22:35.112587294 (213h53m34.983670329s) [upload=187397]: Done starting upload Dec 14 13:22:35.112673899 (213h53m34.983756934s) [upload=187397]: Uploading segments... Dec 14 13:22:35.112767347 (213h53m34.983850393s) [upload=187397]: Got next segment. Inline: false Dec 14 13:22:35.112800511 (213h53m34.983883556s) [upload=187397]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 14 13:22:35.112805821 (213h53m34.983888856s) [upload=187397]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 14 13:22:37.194903810 (213h53m37.065986845s) [seg_pos={0 0}, upload=187397]: Begin upload segment... Dec 14 13:22:37.195008499 (213h53m37.066091534s) [noise=true , address=145.79.227.42:4123 , piece_id=ES2R3XH3RQ2UAJTCWRLZBY3VHUZQVUA2EMCNBYBQSTR3N7NWU6EQ, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.195025751 (213h53m37.066108786s) [seg_pos={0 0}, upload=187397]: Done begin upload segment. Dec 14 13:22:37.195062162 (213h53m37.066145197s) [noise=true , address=145.79.227.44:4193 , piece_id=UMQOWDMKE3PCVS6DQUPMMRR3IYW7EUY3ABOGSFVJ3H7KGFRUZ7HA, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.195125401 (213h53m37.066208446s) [noise=true , address=145.79.227.45:4229 , piece_id=4IHXIUBBXYN6MZYQBRBRMJZ5HYOFNCLHKOVZKFVS6M37QMRE35OA, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.195764098 (213h53m37.066847133s) [noise=true , address=145.79.227.35:4482 , piece_id=RAUYG5T6DQIDOV5QM2OCE4GAPYIFBHUA3ZZR3Y6KN7ZX4OPMPEZA, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.196145995 (213h53m37.06722904s) [noise=true , address=145.79.227.47:4267 , piece_id=FOKTCGQ7VI4GDXRJ64LDLVSALAHAL7GRFI257S6X6G2Y4IGUNW6Q, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.196522081 (213h53m37.067605116s) [noise=true , address=145.79.227.36:4519 , piece_id=5X7PDP7FELCD6A5566SD7YBEUN4WELNB4IDV545B27TJ6DPEIIKA, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.196919738 (213h53m37.068002783s) [noise=true , address=145.79.227.40:4050 , piece_id=OFTY43B5JLDTXZRWXEHQZLUP6YJOL75CAL36EHSOQJR3GIHZOAJQ, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.197734449 (213h53m37.068817484s) [noise=true , address=145.79.227.28:4725 , piece_id=GNFCQWHHQHCCHL7VNU6TYNEFJZFGR7SMXQO66UCSNORVK4XAHPGA, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.197786599 (213h53m37.068869634s) [noise=true , address=145.79.227.43:4158 , piece_id=HDWGVVZAZI3R6KQFIUG74SWRF3G2USEMNKVNMKZDYISPRA74AUGQ, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.198256694 (213h53m37.069339739s) [noise=true , address=145.79.227.35:4481 , piece_id=LTU57UBRXVWPXJKKRNWYSIQHL7YYPNDMI7R5GARIP55AO7KS4TXQ, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.198675992 (213h53m37.069759027s) [noise=true , address=145.79.227.51:4411 , piece_id=V7ISMKJUJ6EDCLYOJIZBCBUUNDIWADZPZUUY7XH2F6BHVZILRXBQ, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.199111532 (213h53m37.070194567s) [noise=true , address=145.79.227.26:4655 , piece_id=NSYNQURKGVD7VTPMXJGF5RDIQKNUBLVKFYUE5CLPQ6IMIEJZQ4SQ, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.199658874 (213h53m37.07074191s) [noise=true , address=145.79.227.34:4905 , piece_id=GHVXBDDAG64LE2A5RRGK3POPOL3QSQGGI7UH3NIHEZTEG7A7GODA, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.200449229 (213h53m37.071532274s) [noise=true , address=145.79.227.38:4978 , piece_id=3AV4CLNBUBIXI73A3YSDKOPIMOROFBI73TVA3KNGDSBSE6KMYNEA, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.200877725 (213h53m37.07196077s) [noise=true , address=145.79.227.46:4601 , piece_id=AFT773CWP62ANAIFPUTDYQ35GSF6K2NFEAO7X4VKIT62CQD3NJEQ, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.207413279 (213h53m37.078496314s) [noise=true , address=145.79.227.31:4797 , piece_id=KRMTNOTWOWCIERSEUWBWZOIPDDUWYZKUBX74ZAIWV737CIKFVRLA, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:39.731884038 (213h53m39.602967073s) [upload=187397]: Next returned nil segment Dec 14 13:22:39.731901430 (213h53m39.602984475s) [upload=187397]: Waiting for error group managing segments... Dec 14 13:22:40.154962424 (213h53m40.026045469s) [noise=true , address=145.79.227.26:4655 , piece_id=NSYNQURKGVD7VTPMXJGF5RDIQKNUBLVKFYUE5CLPQ6IMIEJZQ4SQ, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.159418800 (213h53m40.030501835s) [noise=true , address=145.79.227.40:4050 , piece_id=OFTY43B5JLDTXZRWXEHQZLUP6YJOL75CAL36EHSOQJR3GIHZOAJQ, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.175663783 (213h53m40.046746829s) [noise=true , address=145.79.227.43:4158 , piece_id=HDWGVVZAZI3R6KQFIUG74SWRF3G2USEMNKVNMKZDYISPRA74AUGQ, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.175938396 (213h53m40.047021432s) [noise=true , address=145.79.227.35:4482 , piece_id=RAUYG5T6DQIDOV5QM2OCE4GAPYIFBHUA3ZZR3Y6KN7ZX4OPMPEZA, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.177599980 (213h53m40.048683025s) [noise=true , address=145.79.227.31:4797 , piece_id=KRMTNOTWOWCIERSEUWBWZOIPDDUWYZKUBX74ZAIWV737CIKFVRLA, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.181351754 (213h53m40.052434789s) [noise=true , address=145.79.227.51:4411 , piece_id=V7ISMKJUJ6EDCLYOJIZBCBUUNDIWADZPZUUY7XH2F6BHVZILRXBQ, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.181709967 (213h53m40.052793002s) [noise=true , address=145.79.227.46:4601 , piece_id=AFT773CWP62ANAIFPUTDYQ35GSF6K2NFEAO7X4VKIT62CQD3NJEQ, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.183204752 (213h53m40.054287787s) [noise=true , address=145.79.227.28:4725 , piece_id=GNFCQWHHQHCCHL7VNU6TYNEFJZFGR7SMXQO66UCSNORVK4XAHPGA, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.183813983 (213h53m40.054897028s) [noise=true , address=145.79.227.45:4229 , piece_id=4IHXIUBBXYN6MZYQBRBRMJZ5HYOFNCLHKOVZKFVS6M37QMRE35OA, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.186380270 (213h53m40.057463315s) [noise=true , address=145.79.227.38:4978 , piece_id=3AV4CLNBUBIXI73A3YSDKOPIMOROFBI73TVA3KNGDSBSE6KMYNEA, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.198000036 (213h53m40.069083071s) [noise=true , address=145.79.227.36:4519 , piece_id=5X7PDP7FELCD6A5566SD7YBEUN4WELNB4IDV545B27TJ6DPEIIKA, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.201922084 (213h53m40.07300513s) [noise=true , address=145.79.227.47:4267 , piece_id=FOKTCGQ7VI4GDXRJ64LDLVSALAHAL7GRFI257S6X6G2Y4IGUNW6Q, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.217629133 (213h53m40.088712178s) [noise=true , address=145.79.227.34:4905 , piece_id=GHVXBDDAG64LE2A5RRGK3POPOL3QSQGGI7UH3NIHEZTEG7A7GODA, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.217662106 (213h53m40.088745151s) [seg_pos={0 0}, upload=187397]: Segment reached optimal threshold of 13 pieces. Dec 14 13:22:40.218130388 (213h53m40.089213423s) [noise=true , address=145.79.227.44:4193 , piece_id=UMQOWDMKE3PCVS6DQUPMMRR3IYW7EUY3ABOGSFVJ3H7KGFRUZ7HA, seg_pos={0 0}, upload=187397]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8): protocol: expected piece hash; context canceled; EOF Dec 14 13:22:40.218148793 (213h53m40.089231828s) [noise=true , address=145.79.227.44:4193 , piece_id=UMQOWDMKE3PCVS6DQUPMMRR3IYW7EUY3ABOGSFVJ3H7KGFRUZ7HA, seg_pos={0 0}, upload=187397]: Long tail cancellation due to optimal threshold Dec 14 13:22:40.218265846 (213h53m40.089348891s) [noise=true , address=145.79.227.35:4481 , piece_id=LTU57UBRXVWPXJKKRNWYSIQHL7YYPNDMI7R5GARIP55AO7KS4TXQ, seg_pos={0 0}, upload=187397]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bRK8WWmAnWUwzCfHu8JtZgrruYSJCvhnsYTKcaLccJcv5RQjp): protocol: expected piece hash; context canceled; EOF Dec 14 13:22:40.218275184 (213h53m40.089358239s) [noise=true , address=145.79.227.35:4481 , piece_id=LTU57UBRXVWPXJKKRNWYSIQHL7YYPNDMI7R5GARIP55AO7KS4TXQ, seg_pos={0 0}, upload=187397]: Long tail cancellation due to optimal threshold Dec 14 13:22:40.218315581 (213h53m40.089398616s) [noise=true , address=145.79.227.42:4123 , piece_id=ES2R3XH3RQ2UAJTCWRLZBY3VHUZQVUA2EMCNBYBQSTR3N7NWU6EQ, seg_pos={0 0}, upload=187397]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM): protocol: expected piece hash; context canceled; EOF Dec 14 13:22:40.218330358 (213h53m40.089413393s) [noise=true , address=145.79.227.42:4123 , piece_id=ES2R3XH3RQ2UAJTCWRLZBY3VHUZQVUA2EMCNBYBQSTR3N7NWU6EQ, seg_pos={0 0}, upload=187397]: Long tail cancellation due to optimal threshold Dec 14 13:22:40.218347892 (213h53m40.089430937s) [seg_pos={0 0}, upload=187397]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 14 13:22:40.218452501 (213h53m40.089535556s) [upload=187397]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 14 13:22:40.218479914 (213h53m40.089562969s) [upload=187397]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 14 13:22:43.384827291 (213h53m43.255910336s) [upload=187397]: Done uploading segments... Dec 14 13:22:43.384857198 (213h53m43.255940233s) [upload=187397]: Upload finished. err: <nil> Duration: 14.673187348s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 30.002836727s trace: https://jaeger.dev.storj.io/trace/8ca15e950587d46 FAIL (2025-12-14 13:23:04 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12YKLTah3swX14kL82eNogJxd8D8Hjfq7SPjDDQAapDnXmDctP1, Piece ID: GNFCQWHHQHCCHL7VNU6TYNEFJZFGR7SMXQO66UCSNORVK4XAHPGA) context deadline exceeded; piecestore: (Node ID: 125GxLsL5UtXkuJ1LFvF42N9iZkKtRK3rq63xdR25JQbtwoT1L5, Piece ID: AFT773CWP62ANAIFPUTDYQ35GSF6K2NFEAO7X4VKIT62CQD3NJEQ) context deadline exceeded ======LOGS====== Dec 14 13:22:35.112522281 (213h53m34.983605316s) [upload=187397]: Starting upload Dec 14 13:22:35.112587294 (213h53m34.983670329s) [upload=187397]: Done starting upload Dec 14 13:22:35.112673899 (213h53m34.983756934s) [upload=187397]: Uploading segments... Dec 14 13:22:35.112767347 (213h53m34.983850393s) [upload=187397]: Got next segment. Inline: false Dec 14 13:22:35.112800511 (213h53m34.983883556s) [upload=187397]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 14 13:22:35.112805821 (213h53m34.983888856s) [upload=187397]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 14 13:22:37.194903810 (213h53m37.065986845s) [seg_pos={0 0}, upload=187397]: Begin upload segment... Dec 14 13:22:37.195008499 (213h53m37.066091534s) [noise=true , address=145.79.227.42:4123 , piece_id=ES2R3XH3RQ2UAJTCWRLZBY3VHUZQVUA2EMCNBYBQSTR3N7NWU6EQ, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.195025751 (213h53m37.066108786s) [seg_pos={0 0}, upload=187397]: Done begin upload segment. Dec 14 13:22:37.195062162 (213h53m37.066145197s) [noise=true , address=145.79.227.44:4193 , piece_id=UMQOWDMKE3PCVS6DQUPMMRR3IYW7EUY3ABOGSFVJ3H7KGFRUZ7HA, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.195125401 (213h53m37.066208446s) [noise=true , address=145.79.227.45:4229 , piece_id=4IHXIUBBXYN6MZYQBRBRMJZ5HYOFNCLHKOVZKFVS6M37QMRE35OA, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.195764098 (213h53m37.066847133s) [noise=true , address=145.79.227.35:4482 , piece_id=RAUYG5T6DQIDOV5QM2OCE4GAPYIFBHUA3ZZR3Y6KN7ZX4OPMPEZA, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.196145995 (213h53m37.06722904s) [noise=true , address=145.79.227.47:4267 , piece_id=FOKTCGQ7VI4GDXRJ64LDLVSALAHAL7GRFI257S6X6G2Y4IGUNW6Q, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.196522081 (213h53m37.067605116s) [noise=true , address=145.79.227.36:4519 , piece_id=5X7PDP7FELCD6A5566SD7YBEUN4WELNB4IDV545B27TJ6DPEIIKA, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.196919738 (213h53m37.068002783s) [noise=true , address=145.79.227.40:4050 , piece_id=OFTY43B5JLDTXZRWXEHQZLUP6YJOL75CAL36EHSOQJR3GIHZOAJQ, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.197734449 (213h53m37.068817484s) [noise=true , address=145.79.227.28:4725 , piece_id=GNFCQWHHQHCCHL7VNU6TYNEFJZFGR7SMXQO66UCSNORVK4XAHPGA, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.197786599 (213h53m37.068869634s) [noise=true , address=145.79.227.43:4158 , piece_id=HDWGVVZAZI3R6KQFIUG74SWRF3G2USEMNKVNMKZDYISPRA74AUGQ, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.198256694 (213h53m37.069339739s) [noise=true , address=145.79.227.35:4481 , piece_id=LTU57UBRXVWPXJKKRNWYSIQHL7YYPNDMI7R5GARIP55AO7KS4TXQ, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.198675992 (213h53m37.069759027s) [noise=true , address=145.79.227.51:4411 , piece_id=V7ISMKJUJ6EDCLYOJIZBCBUUNDIWADZPZUUY7XH2F6BHVZILRXBQ, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.199111532 (213h53m37.070194567s) [noise=true , address=145.79.227.26:4655 , piece_id=NSYNQURKGVD7VTPMXJGF5RDIQKNUBLVKFYUE5CLPQ6IMIEJZQ4SQ, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.199658874 (213h53m37.07074191s) [noise=true , address=145.79.227.34:4905 , piece_id=GHVXBDDAG64LE2A5RRGK3POPOL3QSQGGI7UH3NIHEZTEG7A7GODA, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.200449229 (213h53m37.071532274s) [noise=true , address=145.79.227.38:4978 , piece_id=3AV4CLNBUBIXI73A3YSDKOPIMOROFBI73TVA3KNGDSBSE6KMYNEA, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.200877725 (213h53m37.07196077s) [noise=true , address=145.79.227.46:4601 , piece_id=AFT773CWP62ANAIFPUTDYQ35GSF6K2NFEAO7X4VKIT62CQD3NJEQ, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:37.207413279 (213h53m37.078496314s) [noise=true , address=145.79.227.31:4797 , piece_id=KRMTNOTWOWCIERSEUWBWZOIPDDUWYZKUBX74ZAIWV737CIKFVRLA, seg_pos={0 0}, upload=187397]: Uploading piece... Dec 14 13:22:39.731884038 (213h53m39.602967073s) [upload=187397]: Next returned nil segment Dec 14 13:22:39.731901430 (213h53m39.602984475s) [upload=187397]: Waiting for error group managing segments... Dec 14 13:22:40.154962424 (213h53m40.026045469s) [noise=true , address=145.79.227.26:4655 , piece_id=NSYNQURKGVD7VTPMXJGF5RDIQKNUBLVKFYUE5CLPQ6IMIEJZQ4SQ, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.159418800 (213h53m40.030501835s) [noise=true , address=145.79.227.40:4050 , piece_id=OFTY43B5JLDTXZRWXEHQZLUP6YJOL75CAL36EHSOQJR3GIHZOAJQ, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.175663783 (213h53m40.046746829s) [noise=true , address=145.79.227.43:4158 , piece_id=HDWGVVZAZI3R6KQFIUG74SWRF3G2USEMNKVNMKZDYISPRA74AUGQ, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.175938396 (213h53m40.047021432s) [noise=true , address=145.79.227.35:4482 , piece_id=RAUYG5T6DQIDOV5QM2OCE4GAPYIFBHUA3ZZR3Y6KN7ZX4OPMPEZA, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.177599980 (213h53m40.048683025s) [noise=true , address=145.79.227.31:4797 , piece_id=KRMTNOTWOWCIERSEUWBWZOIPDDUWYZKUBX74ZAIWV737CIKFVRLA, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.181351754 (213h53m40.052434789s) [noise=true , address=145.79.227.51:4411 , piece_id=V7ISMKJUJ6EDCLYOJIZBCBUUNDIWADZPZUUY7XH2F6BHVZILRXBQ, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.181709967 (213h53m40.052793002s) [noise=true , address=145.79.227.46:4601 , piece_id=AFT773CWP62ANAIFPUTDYQ35GSF6K2NFEAO7X4VKIT62CQD3NJEQ, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.183204752 (213h53m40.054287787s) [noise=true , address=145.79.227.28:4725 , piece_id=GNFCQWHHQHCCHL7VNU6TYNEFJZFGR7SMXQO66UCSNORVK4XAHPGA, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.183813983 (213h53m40.054897028s) [noise=true , address=145.79.227.45:4229 , piece_id=4IHXIUBBXYN6MZYQBRBRMJZ5HYOFNCLHKOVZKFVS6M37QMRE35OA, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.186380270 (213h53m40.057463315s) [noise=true , address=145.79.227.38:4978 , piece_id=3AV4CLNBUBIXI73A3YSDKOPIMOROFBI73TVA3KNGDSBSE6KMYNEA, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.198000036 (213h53m40.069083071s) [noise=true , address=145.79.227.36:4519 , piece_id=5X7PDP7FELCD6A5566SD7YBEUN4WELNB4IDV545B27TJ6DPEIIKA, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.201922084 (213h53m40.07300513s) [noise=true , address=145.79.227.47:4267 , piece_id=FOKTCGQ7VI4GDXRJ64LDLVSALAHAL7GRFI257S6X6G2Y4IGUNW6Q, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.217629133 (213h53m40.088712178s) [noise=true , address=145.79.227.34:4905 , piece_id=GHVXBDDAG64LE2A5RRGK3POPOL3QSQGGI7UH3NIHEZTEG7A7GODA, seg_pos={0 0}, upload=187397]: Done uploading piece. err: <nil> Dec 14 13:22:40.217662106 (213h53m40.088745151s) [seg_pos={0 0}, upload=187397]: Segment reached optimal threshold of 13 pieces. Dec 14 13:22:40.218130388 (213h53m40.089213423s) [noise=true , address=145.79.227.44:4193 , piece_id=UMQOWDMKE3PCVS6DQUPMMRR3IYW7EUY3ABOGSFVJ3H7KGFRUZ7HA, seg_pos={0 0}, upload=187397]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8): protocol: expected piece hash; context canceled; EOF Dec 14 13:22:40.218148793 (213h53m40.089231828s) [noise=true , address=145.79.227.44:4193 , piece_id=UMQOWDMKE3PCVS6DQUPMMRR3IYW7EUY3ABOGSFVJ3H7KGFRUZ7HA, seg_pos={0 0}, upload=187397]: Long tail cancellation due to optimal threshold Dec 14 13:22:40.218265846 (213h53m40.089348891s) [noise=true , address=145.79.227.35:4481 , piece_id=LTU57UBRXVWPXJKKRNWYSIQHL7YYPNDMI7R5GARIP55AO7KS4TXQ, seg_pos={0 0}, upload=187397]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bRK8WWmAnWUwzCfHu8JtZgrruYSJCvhnsYTKcaLccJcv5RQjp): protocol: expected piece hash; context canceled; EOF Dec 14 13:22:40.218275184 (213h53m40.089358239s) [noise=true , address=145.79.227.35:4481 , piece_id=LTU57UBRXVWPXJKKRNWYSIQHL7YYPNDMI7R5GARIP55AO7KS4TXQ, seg_pos={0 0}, upload=187397]: Long tail cancellation due to optimal threshold Dec 14 13:22:40.218315581 (213h53m40.089398616s) [noise=true , address=145.79.227.42:4123 , piece_id=ES2R3XH3RQ2UAJTCWRLZBY3VHUZQVUA2EMCNBYBQSTR3N7NWU6EQ, seg_pos={0 0}, upload=187397]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM): protocol: expected piece hash; context canceled; EOF Dec 14 13:22:40.218330358 (213h53m40.089413393s) [noise=true , address=145.79.227.42:4123 , piece_id=ES2R3XH3RQ2UAJTCWRLZBY3VHUZQVUA2EMCNBYBQSTR3N7NWU6EQ, seg_pos={0 0}, upload=187397]: Long tail cancellation due to optimal threshold Dec 14 13:22:40.218347892 (213h53m40.089430937s) [seg_pos={0 0}, upload=187397]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 14 13:22:40.218452501 (213h53m40.089535556s) [upload=187397]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 14 13:22:40.218479914 (213h53m40.089562969s) [upload=187397]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 14 13:22:43.384827291 (213h53m43.255910336s) [upload=187397]: Done uploading segments... Dec 14 13:22:43.384857198 (213h53m43.255940233s) [upload=187397]: Upload finished. err: <nil> Duration: 30.002859671s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [............X................X..............................] duration: 16.322466066s trace: https://jaeger.dev.storj.io/trace/1c9f407bc6620bcf FAIL (2025-12-14 13:23:03 UTC) orbiter warning: context deadline exceeded Duration: 16.322481566s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.............X................X.............................] duration: 28.287435689s trace: https://jaeger.dev.storj.io/trace/1c9f407bc6620bcf FAIL (2025-12-14 13:22:46 UTC) orbiter warning: context deadline exceeded Duration: 28.287460246s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..............X................X............................] duration: 11.766792017s trace: https://jaeger.dev.storj.io/trace/1c9f407bc6620bcf FAIL (2025-12-14 13:21:27 UTC) orbiter warning: context deadline exceeded Duration: 11.766820231s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...............X................X...........................] duration: 18.840660447s trace: https://jaeger.dev.storj.io/trace/1c9f407bc6620bcf FAIL (2025-12-14 13:21:07 UTC) orbiter warning: context deadline exceeded Duration: 18.840687688s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [................X................X..........................] duration: 25.848784359s trace: https://jaeger.dev.storj.io/trace/1c9f407bc6620bcf FAIL (2025-12-14 13:20:29 UTC) orbiter warning: context deadline exceeded Duration: 25.848818104s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [X...........................................................] duration: 3.558426507s trace: https://jaeger.dev.storj.io/trace/2e557986b2c34ff1 FAIL (2025-12-14 13:20:27 UTC) orbiter warning: context deadline exceeded Duration: 3.558456433s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.X..........................................................] duration: 3.616368035s trace: https://jaeger.dev.storj.io/trace/2e557986b2c34ff1 FAIL (2025-12-14 13:19:14 UTC) orbiter warning: context deadline exceeded Duration: 3.616400406s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.................X................X.........................] duration: 18.505764706s trace: https://jaeger.dev.storj.io/trace/1c9f407bc6620bcf FAIL (2025-12-14 13:19:11 UTC) orbiter warning: context deadline exceeded Duration: 18.505781207s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..X.........................................................] duration: 4.79499141s trace: https://jaeger.dev.storj.io/trace/2e557986b2c34ff1 FAIL (2025-12-14 13:18:56 UTC) orbiter warning: context deadline exceeded Duration: 4.795024253s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..................X................X........................] duration: 20.003395934s trace: https://jaeger.dev.storj.io/trace/1c9f407bc6620bcf FAIL (2025-12-14 13:18:52 UTC) orbiter warning: context deadline exceeded Duration: 20.003447403s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...X........................................................] duration: 4.499113922s trace: https://jaeger.dev.storj.io/trace/2e557986b2c34ff1 FAIL (2025-12-14 13:18:22 UTC) orbiter warning: context deadline exceeded Duration: 4.499143138s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [....X.......................................................] duration: 4.289518577s trace: https://jaeger.dev.storj.io/trace/2e557986b2c34ff1 FAIL (2025-12-14 13:17:49 UTC) orbiter warning: context deadline exceeded Duration: 4.289543305s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...................X................X.......................] duration: 10.119652392s trace: https://jaeger.dev.storj.io/trace/1c9f407bc6620bcf FAIL (2025-12-14 13:17:48 UTC) orbiter warning: context deadline exceeded Duration: 10.119694713s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.....X......................................................] duration: 4.808238522s trace: https://jaeger.dev.storj.io/trace/2e557986b2c34ff1 FAIL (2025-12-14 13:17:33 UTC) orbiter warning: context deadline exceeded Duration: 4.808269272s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [....................X................X......................] duration: 20.994875245s trace: https://jaeger.dev.storj.io/trace/1c9f407bc6620bcf FAIL (2025-12-14 13:17:24 UTC) orbiter warning: context deadline exceeded Duration: 20.994906665s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [......X.....................................................] duration: 4.117489885s trace: https://jaeger.dev.storj.io/trace/2e557986b2c34ff1 FAIL (2025-12-14 13:16:57 UTC) orbiter warning: context deadline exceeded Duration: 4.117518791s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.......X....................................................] duration: 4.009376923s trace: https://jaeger.dev.storj.io/trace/2e557986b2c34ff1 FAIL (2025-12-14 13:16:33 UTC) orbiter warning: context deadline exceeded Duration: 4.00939587s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....................X................X.....................] duration: 18.954967864s trace: https://jaeger.dev.storj.io/trace/1c9f407bc6620bcf FAIL (2025-12-14 13:16:19 UTC) orbiter warning: context deadline exceeded Duration: 18.955001557s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [........X...................................................] duration: 4.561773732s trace: https://jaeger.dev.storj.io/trace/2e557986b2c34ff1 FAIL (2025-12-14 13:16:13 UTC) orbiter warning: context deadline exceeded Duration: 4.561802927s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [......................X................X....................] duration: 12.202264478s trace: https://jaeger.dev.storj.io/trace/1c9f407bc6620bcf FAIL (2025-12-14 13:15:26 UTC) orbiter warning: context deadline exceeded Duration: 12.202290267s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.........X..................................................] duration: 4.652386905s trace: https://jaeger.dev.storj.io/trace/2e557986b2c34ff1 FAIL (2025-12-14 13:15:10 UTC) orbiter warning: context deadline exceeded Duration: 4.652512984s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......................X................X...................] duration: 22.459485179s trace: https://jaeger.dev.storj.io/trace/1c9f407bc6620bcf FAIL (2025-12-14 13:14:51 UTC) orbiter warning: context deadline exceeded Duration: 22.459513814s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [X.........X.................................................] duration: 3.660392936s trace: https://jaeger.dev.storj.io/trace/2e557986b2c34ff1 FAIL (2025-12-14 13:14:15 UTC) orbiter warning: context deadline exceeded Duration: 3.660420169s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [........................X................X..................] duration: 18.18002234s trace: https://jaeger.dev.storj.io/trace/1c9f407bc6620bcf FAIL (2025-12-14 13:13:51 UTC) orbiter warning: context deadline exceeded Duration: 18.180055683s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [.X.........X................................................] duration: 3.934410312s trace: https://jaeger.dev.storj.io/trace/2e557986b2c34ff1 FAIL (2025-12-14 13:13:34 UTC) orbiter warning: context deadline exceeded Duration: 3.934434328s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.........................X................X.................] duration: 21.346013606s trace: https://jaeger.dev.storj.io/trace/1c9f407bc6620bcf FAIL (2025-12-14 13:13:25 UTC) orbiter warning: context deadline exceeded Duration: 21.346043334s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..........................X................X................] duration: 20.711195506s trace: https://jaeger.dev.storj.io/trace/1c9f407bc6620bcf FAIL (2025-12-14 13:13:00 UTC) orbiter warning: context deadline exceeded Duration: 20.71122338s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [X..........................X................X...............] duration: 19.23991543s trace: https://jaeger.dev.storj.io/trace/1c9f407bc6620bcf FAIL (2025-12-14 13:12:26 UTC) orbiter warning: context deadline exceeded Duration: 19.239948221s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [..X.........X...............................................] duration: 3.677790726s trace: https://jaeger.dev.storj.io/trace/2e557986b2c34ff1 FAIL (2025-12-14 13:12:20 UTC) orbiter warning: context deadline exceeded Duration: 3.677817318s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)