Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [......X...X...........X........X..X....XX.XXX....X.......XX.] duration: 21.413049745s trace: https://jaeger.dev.storj.io/trace/6c07707c87e1fb63 FAIL (2025-12-31 00:46:10 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, Piece ID: GG4MQX6GQGCO54R7B2FQFKF3AVDCV6BGGU7GMDAVNYWJ23AESLGQ) context deadline exceeded; piecestore: (Node ID: 12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f, Piece ID: 7QVQHE6NXP7GGAHH5AYHXGL53OML2SVEYN42XXFUEZFMAI3GZDZA) context deadline exceeded; piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: CCLTLM57NIXBAAODKCOZ4YVX37CDV33A7A3WYR2PSSHEIKO2RQ2Q) context deadline exceeded; piecestore: (Node ID: 12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, Piece ID: QHK56YLQYVIN4LADYSB3YKE6TF3LTDXCEZOGH4KE4FVC2A2W42CA) context deadline exceeded ======LOGS====== Dec 31 00:44:45.443637561 (609h15m45.314720596s) [upload=533955]: Starting upload Dec 31 00:44:45.443709788 (609h15m45.314792833s) [upload=533955]: Done starting upload Dec 31 00:44:45.443796563 (609h15m45.314879608s) [upload=533955]: Uploading segments... Dec 31 00:44:45.443910320 (609h15m45.314993355s) [upload=533955]: Got next segment. Inline: false Dec 31 00:44:45.443944105 (609h15m45.31502714s) [upload=533955]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 00:44:45.443948483 (609h15m45.315031518s) [upload=533955]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 00:44:45.691936380 (609h15m45.563019415s) [seg_pos={0 0}, upload=533955]: Begin upload segment... Dec 31 00:44:45.691985453 (609h15m45.563068488s) [seg_pos={0 0}, upload=533955]: Done begin upload segment. Dec 31 00:44:45.692065505 (609h15m45.56314854s) [noise=true , address=145.79.227.44:4194 , piece_id=QYEHIRT3MVMVKM6YWOW7DL7QPY5UPXJCRVEOLMIS4N4JGYOXCG7A, seg_pos={0 0}, upload=533955]: Uploading piece... Dec 31 00:44:45.692086035 (609h15m45.56316907s) [noise=true , address=145.79.227.32:4834 , piece_id=QX55FO4R3LZHFC2QBFRMZLAMGXGDSUH2TH4FD7BX5D3RR6BSXSQQ, seg_pos={0 0}, upload=533955]: Uploading piece... Dec 31 00:44:45.692196725 (609h15m45.56327976s) [noise=true , address=45.91.116.6:4975 , piece_id=CCLTLM57NIXBAAODKCOZ4YVX37CDV33A7A3WYR2PSSHEIKO2RQ2Q, seg_pos={0 0}, upload=533955]: Uploading piece... Dec 31 00:44:45.692602398 (609h15m45.563685433s) [noise=true , address=77.90.31.11:4555 , piece_id=D7HD3K56K6GLAKLIA4AHV4FBIOICZVKO467WYG3VI6YD7WBAWU6Q, seg_pos={0 0}, upload=533955]: Uploading piece... Dec 31 00:44:45.692612487 (609h15m45.563695522s) [noise=true , address=45.91.116.4:4940 , piece_id=QVIN7MZ5TO2DI257G4RXFIRASHBCUT7J327MDWMOLRSHGO4OTP6Q, seg_pos={0 0}, upload=533955]: Uploading piece... Dec 31 00:44:45.692917979 (609h15m45.564001014s) [noise=true , address=145.79.227.21:4475 , piece_id=DDI4OGAJOIDAY5QWK33H5YA6VWSKXET2MSFDQ7VLC6UQ7WDGSRLQ, seg_pos={0 0}, upload=533955]: Uploading piece... Dec 31 00:44:45.693173665 (609h15m45.564256751s) [noise=true , address=145.79.227.19:4078 , piece_id=PTQY2TGFMUEBJ2PGB5WWW6ICT453L7CXY322TMS6655Z5UJ6GBVA, seg_pos={0 0}, upload=533955]: Uploading piece... Dec 31 00:44:45.693253016 (609h15m45.564336061s) [noise=true , address=145.79.227.2:4115 , piece_id=XBWADCTRARLWVQWHZ3IVMXDN2FYEE34WWZ5YFQE24BM7VGTRPNIA, seg_pos={0 0}, upload=533955]: Uploading piece... Dec 31 00:44:45.693476122 (609h15m45.564559157s) [noise=true , address=45.91.116.12:4227 , piece_id=QHK56YLQYVIN4LADYSB3YKE6TF3LTDXCEZOGH4KE4FVC2A2W42CA, seg_pos={0 0}, upload=533955]: Uploading piece... Dec 31 00:44:45.693772385 (609h15m45.56485542s) [noise=true , address=45.91.116.7:4009 , piece_id=BTIV67EFKX5OFXBIIGGS2GDXMUHR3EHG4S6ZMI3LQFSPLV4PVR5A, seg_pos={0 0}, upload=533955]: Uploading piece... Dec 31 00:44:45.694100911 (609h15m45.565183946s) [noise=true , address=145.79.227.22:4511 , piece_id=C2R77ZH5L72IWZEOZSYQOVIH4MD7Z2MTXVK7EF6IWH5DEUPOLW7Q, seg_pos={0 0}, upload=533955]: Uploading piece... Dec 31 00:44:45.694395382 (609h15m45.565478417s) [noise=true , address=145.79.227.13:4802 , piece_id=TXFKE7MTYB5NNC762QORY3UDB5Q42HJ2J4JEKQZIXIUFQYQBFZ5Q, seg_pos={0 0}, upload=533955]: Uploading piece... Dec 31 00:44:45.694842984 (609h15m45.565926019s) [noise=true , address=45.91.116.5:4298 , piece_id=GG4MQX6GQGCO54R7B2FQFKF3AVDCV6BGGU7GMDAVNYWJ23AESLGQ, seg_pos={0 0}, upload=533955]: Uploading piece... Dec 31 00:44:45.695220373 (609h15m45.566303408s) [noise=true , address=145.79.227.23:4546 , piece_id=5RMH6VUR7LQBHGNYAE7R4O4GF3GGHN5ZVL7Q6C6V3YJ4HGFUS3YA, seg_pos={0 0}, upload=533955]: Uploading piece... Dec 31 00:44:45.696046176 (609h15m45.567129211s) [noise=true , address=145.79.227.5:4222 , piece_id=I55PDNWS2YXLSQGEW23FNSXQQQDANQBHAR2T2OH2V4CCA2P7OHFQ, seg_pos={0 0}, upload=533955]: Uploading piece... Dec 31 00:44:45.697349988 (609h15m45.568433023s) [noise=true , address=77.90.31.13:4629 , piece_id=7QVQHE6NXP7GGAHH5AYHXGL53OML2SVEYN42XXFUEZFMAI3GZDZA, seg_pos={0 0}, upload=533955]: Uploading piece... Dec 31 00:44:47.389082715 (609h15m47.26016575s) [upload=533955]: Next returned nil segment Dec 31 00:44:47.389100959 (609h15m47.260183994s) [upload=533955]: Waiting for error group managing segments... Dec 31 00:44:47.679637866 (609h15m47.550720911s) [noise=true , address=145.79.227.22:4511 , piece_id=C2R77ZH5L72IWZEOZSYQOVIH4MD7Z2MTXVK7EF6IWH5DEUPOLW7Q, seg_pos={0 0}, upload=533955]: Done uploading piece. err: <nil> Dec 31 00:44:47.681072347 (609h15m47.552155382s) [noise=true , address=145.79.227.23:4546 , piece_id=5RMH6VUR7LQBHGNYAE7R4O4GF3GGHN5ZVL7Q6C6V3YJ4HGFUS3YA, seg_pos={0 0}, upload=533955]: Done uploading piece. err: <nil> Dec 31 00:44:47.681636803 (609h15m47.552719838s) [noise=true , address=145.79.227.21:4475 , piece_id=DDI4OGAJOIDAY5QWK33H5YA6VWSKXET2MSFDQ7VLC6UQ7WDGSRLQ, seg_pos={0 0}, upload=533955]: Done uploading piece. err: <nil> Dec 31 00:44:47.682153145 (609h15m47.5532362s) [noise=true , address=145.79.227.2:4115 , piece_id=XBWADCTRARLWVQWHZ3IVMXDN2FYEE34WWZ5YFQE24BM7VGTRPNIA, seg_pos={0 0}, upload=533955]: Done uploading piece. err: <nil> Dec 31 00:44:47.684310152 (609h15m47.555393207s) [noise=true , address=145.79.227.32:4834 , piece_id=QX55FO4R3LZHFC2QBFRMZLAMGXGDSUH2TH4FD7BX5D3RR6BSXSQQ, seg_pos={0 0}, upload=533955]: Done uploading piece. err: <nil> Dec 31 00:44:47.693448863 (609h15m47.564531909s) [noise=true , address=145.79.227.13:4802 , piece_id=TXFKE7MTYB5NNC762QORY3UDB5Q42HJ2J4JEKQZIXIUFQYQBFZ5Q, seg_pos={0 0}, upload=533955]: Done uploading piece. err: <nil> Dec 31 00:44:47.697022899 (609h15m47.568105944s) [noise=true , address=145.79.227.5:4222 , piece_id=I55PDNWS2YXLSQGEW23FNSXQQQDANQBHAR2T2OH2V4CCA2P7OHFQ, seg_pos={0 0}, upload=533955]: Done uploading piece. err: <nil> Dec 31 00:44:47.761759876 (609h15m47.632842911s) [noise=true , address=145.79.227.19:4078 , piece_id=PTQY2TGFMUEBJ2PGB5WWW6ICT453L7CXY322TMS6655Z5UJ6GBVA, seg_pos={0 0}, upload=533955]: Done uploading piece. err: <nil> Dec 31 00:44:48.007827617 (609h15m47.878910662s) [noise=true , address=45.91.116.4:4940 , piece_id=QVIN7MZ5TO2DI257G4RXFIRASHBCUT7J327MDWMOLRSHGO4OTP6Q, seg_pos={0 0}, upload=533955]: Done uploading piece. err: <nil> Dec 31 00:44:48.012739720 (609h15m47.883822765s) [noise=true , address=45.91.116.6:4975 , piece_id=CCLTLM57NIXBAAODKCOZ4YVX37CDV33A7A3WYR2PSSHEIKO2RQ2Q, seg_pos={0 0}, upload=533955]: Done uploading piece. err: <nil> Dec 31 00:44:48.026107714 (609h15m47.897190749s) [noise=true , address=45.91.116.5:4298 , piece_id=GG4MQX6GQGCO54R7B2FQFKF3AVDCV6BGGU7GMDAVNYWJ23AESLGQ, seg_pos={0 0}, upload=533955]: Done uploading piece. err: <nil> Dec 31 00:44:48.186776747 (609h15m48.057859792s) [noise=true , address=77.90.31.13:4629 , piece_id=7QVQHE6NXP7GGAHH5AYHXGL53OML2SVEYN42XXFUEZFMAI3GZDZA, seg_pos={0 0}, upload=533955]: Done uploading piece. err: <nil> Dec 31 00:44:48.360970714 (609h15m48.232053749s) [noise=true , address=45.91.116.12:4227 , piece_id=QHK56YLQYVIN4LADYSB3YKE6TF3LTDXCEZOGH4KE4FVC2A2W42CA, seg_pos={0 0}, upload=533955]: Done uploading piece. err: <nil> Dec 31 00:44:48.360990682 (609h15m48.232073727s) [seg_pos={0 0}, upload=533955]: Segment reached optimal threshold of 13 pieces. Dec 31 00:44:48.361159894 (609h15m48.232242939s) [noise=true , address=45.91.116.7:4009 , piece_id=BTIV67EFKX5OFXBIIGGS2GDXMUHR3EHG4S6ZMI3LQFSPLV4PVR5A, seg_pos={0 0}, upload=533955]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: write tcp 188.166.225.170:52828->45.91.116.7:4009: use of closed network connection; write tcp 188.166.225.170:52828->45.91.116.7:4009: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52828->45.91.116.7:4009: use of closed network connection Dec 31 00:44:48.361165144 (609h15m48.232248189s) [noise=true , address=45.91.116.7:4009 , piece_id=BTIV67EFKX5OFXBIIGGS2GDXMUHR3EHG4S6ZMI3LQFSPLV4PVR5A, seg_pos={0 0}, upload=533955]: Long tail cancellation due to optimal threshold Dec 31 00:44:48.361331891 (609h15m48.232414946s) [noise=true , address=145.79.227.44:4194 , piece_id=QYEHIRT3MVMVKM6YWOW7DL7QPY5UPXJCRVEOLMIS4N4JGYOXCG7A, seg_pos={0 0}, upload=533955]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh): protocol: expected piece hash; context canceled; EOF Dec 31 00:44:48.361343514 (609h15m48.232426559s) [noise=true , address=145.79.227.44:4194 , piece_id=QYEHIRT3MVMVKM6YWOW7DL7QPY5UPXJCRVEOLMIS4N4JGYOXCG7A, seg_pos={0 0}, upload=533955]: Long tail cancellation due to optimal threshold Dec 31 00:44:48.361388970 (609h15m48.232472005s) [noise=true , address=77.90.31.11:4555 , piece_id=D7HD3K56K6GLAKLIA4AHV4FBIOICZVKO467WYG3VI6YD7WBAWU6Q, seg_pos={0 0}, upload=533955]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ): protocol: expected piece hash; context canceled; EOF Dec 31 00:44:48.361395132 (609h15m48.232478167s) [noise=true , address=77.90.31.11:4555 , piece_id=D7HD3K56K6GLAKLIA4AHV4FBIOICZVKO467WYG3VI6YD7WBAWU6Q, seg_pos={0 0}, upload=533955]: Long tail cancellation due to optimal threshold Dec 31 00:44:48.361412014 (609h15m48.232495049s) [seg_pos={0 0}, upload=533955]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 00:44:48.361474594 (609h15m48.232557629s) [upload=533955]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 00:44:48.361479443 (609h15m48.232562478s) [upload=533955]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 00:44:48.655948215 (609h15m48.52703125s) [upload=533955]: Done uploading segments... Dec 31 00:44:48.655991958 (609h15m48.527074993s) [upload=533955]: Upload finished. err: <nil> Duration: 21.413071095s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [....................................................X.......] duration: 3.416446602s trace: https://jaeger.dev.storj.io/trace/2c1d476356ae4e72 FAIL (2025-12-31 00:45:52 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.416504613s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 15.802306537s trace: https://jaeger.dev.storj.io/trace/67f6164cdd15ab81 FAIL (2025-12-31 00:45:47 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== 31 00:33:25.026367244 (609h4m24.897450289s) [noise=true , address=145.79.227.2:4128 , piece_id=DW37Q32FFAZ37KHMOII6XGCJVDO2XTKED5TLOKG65JULWTXJIQEA, seg_pos={0 0}, upload=533792]: Uploading piece... Dec 31 00:33:25.026878247 (609h4m24.897961292s) [noise=true , address=145.79.227.44:4217 , piece_id=GW44B36EHWVCRLEM6BA5MXC3ITRNCCJATW2RXSYTNC7K7WRO2IDQ, seg_pos={0 0}, upload=533792]: Uploading piece... Dec 31 00:33:25.027303136 (609h4m24.898386171s) [noise=true , address=45.91.116.13:4161 , piece_id=K4CTNQGHL5S2GOARW2F42REGZ4QVWA2RKUNULONYX3VPVHAGONUQ, seg_pos={0 0}, upload=533792]: Uploading piece... Dec 31 00:33:25.027901605 (609h4m24.89898465s) [noise=true , address=77.90.31.13:4632 , piece_id=5KUEGWEJV5NJE2EX3S5S2G7TS5TK23ZQXV3WPA4LLHO65QJRVW6Q, seg_pos={0 0}, upload=533792]: Uploading piece... Dec 31 00:33:25.028257373 (609h4m24.899340408s) [noise=true , address=145.79.227.30:4769 , piece_id=P2ANRKDKCOTOMCZZMDIH2YRVI366UOHWO3LXV7INEDB2GW35Y52A, seg_pos={0 0}, upload=533792]: Uploading piece... Dec 31 00:33:25.028623710 (609h4m24.899706745s) [noise=true , address=145.79.227.34:4930 , piece_id=L3AVRBWPDFTZANZKO4TGITCA3WHBECKEQ2BGMLFQJYN4CR7FSADA, seg_pos={0 0}, upload=533792]: Uploading piece... Dec 31 00:33:25.028952256 (609h4m24.900035291s) [noise=true , address=145.79.227.25:4630 , piece_id=5BVJWWAW2QUQABWWZCD4EW54ZVH3K5RJNM5EXJJUGXI33O6CC24A, seg_pos={0 0}, upload=533792]: Uploading piece... Dec 31 00:33:25.029314766 (609h4m24.900397801s) [noise=true , address=145.79.227.29:4971 , piece_id=EIY6YRG5ZRMEW7IWE5X25RCTGNCU3HQ27O7Q434OAZ4EI335HU4A, seg_pos={0 0}, upload=533792]: Uploading piece... Dec 31 00:33:25.029659533 (609h4m24.900742578s) [noise=true , address=145.79.227.6:4283 , piece_id=MXSG7TVTQBR2AFONVXYWQB4DBAQEEXW226FDC6SSVG5TTWGPAWFQ, seg_pos={0 0}, upload=533792]: Uploading piece... Dec 31 00:33:25.030576118 (609h4m24.901659153s) [noise=true , address=77.90.31.18:4835 , piece_id=GRMJTN6ZDDYITHJWX5UGIAGLCOW6AKADHTK6UCVGRB437TUTDFGA, seg_pos={0 0}, upload=533792]: Uploading piece... Dec 31 00:33:25.031413452 (609h4m24.902496487s) [noise=true , address=64.49.13.155:4913 , piece_id=E4WJCMYJVURPHENWEWR46KOAUAOZU7FW7WLC53DAU44BE7R3CHMQ, seg_pos={0 0}, upload=533792]: Uploading piece... Dec 31 00:33:25.031987204 (609h4m24.903070239s) [noise=true , address=145.79.227.49:4370 , piece_id=52YCNJY7BLJ3N3HDWBZCM7PP2ZPHHBS4GV4QS4WM2AWSLYKB5BBA, seg_pos={0 0}, upload=533792]: Uploading piece... Dec 31 00:33:25.032849947 (609h4m24.903932993s) [noise=true , address=145.79.227.4:4189 , piece_id=ZZF4DSIPQBWKBD45UR2J5P75U5VVMTVB54NEX5QKZV4K2WJZ43BQ, seg_pos={0 0}, upload=533792]: Uploading piece... Dec 31 00:33:25.034222320 (609h4m24.905305355s) [noise=true , address=145.79.227.49:4353 , piece_id=MVTAW6QPJY2LRNEGISHNDP7AC5ZIIUR5N5BBZRFO5I6VYJU7UR3Q, seg_pos={0 0}, upload=533792]: Uploading piece... Dec 31 00:33:25.035359376 (609h4m24.906442411s) [noise=true , address=145.79.227.35:4494 , piece_id=EY2OJLQIF4CH7DQWJWXSBJMQI37NAXSBLQP4ZPMU3IBRVLWJVYWA, seg_pos={0 0}, upload=533792]: Uploading piece... Dec 31 00:33:25.035852344 (609h4m24.906935379s) [noise=true , address=145.79.227.33:4873 , piece_id=2DSEGSC5UWL76NEXSE7WXOGSQNRNXALAINY52TLQHB57LTS2R6ZQ, seg_pos={0 0}, upload=533792]: Uploading piece... Dec 31 00:33:26.491429069 (609h4m26.362512104s) [upload=533792]: Next returned nil segment Dec 31 00:33:26.491441904 (609h4m26.362524939s) [upload=533792]: Waiting for error group managing segments... Dec 31 00:33:26.876786140 (609h4m26.747869175s) [noise=true , address=145.79.227.6:4283 , piece_id=MXSG7TVTQBR2AFONVXYWQB4DBAQEEXW226FDC6SSVG5TTWGPAWFQ, seg_pos={0 0}, upload=533792]: Done uploading piece. err: <nil> Dec 31 00:33:26.895923271 (609h4m26.767006316s) [noise=true , address=145.79.227.35:4494 , piece_id=EY2OJLQIF4CH7DQWJWXSBJMQI37NAXSBLQP4ZPMU3IBRVLWJVYWA, seg_pos={0 0}, upload=533792]: Done uploading piece. err: <nil> Dec 31 00:33:26.903106888 (609h4m26.774189923s) [noise=true , address=145.79.227.33:4873 , piece_id=2DSEGSC5UWL76NEXSE7WXOGSQNRNXALAINY52TLQHB57LTS2R6ZQ, seg_pos={0 0}, upload=533792]: Done uploading piece. err: <nil> Dec 31 00:33:26.912484285 (609h4m26.78356733s) [noise=true , address=145.79.227.29:4971 , piece_id=EIY6YRG5ZRMEW7IWE5X25RCTGNCU3HQ27O7Q434OAZ4EI335HU4A, seg_pos={0 0}, upload=533792]: Done uploading piece. err: <nil> Dec 31 00:33:27.004280888 (609h4m26.875363933s) [noise=true , address=145.79.227.21:4495 , piece_id=YWETKWGBGTY3LALK4DXWSKUTM2TFEYB4N5A3KNKW346V3F5XRN2Q, seg_pos={0 0}, upload=533792]: Done uploading piece. err: <nil> Dec 31 00:33:27.118759862 (609h4m26.989842897s) [noise=true , address=77.90.31.13:4632 , piece_id=5KUEGWEJV5NJE2EX3S5S2G7TS5TK23ZQXV3WPA4LLHO65QJRVW6Q, seg_pos={0 0}, upload=533792]: Done uploading piece. err: <nil> Dec 31 00:33:27.225324635 (609h4m27.09640767s) [noise=true , address=45.91.116.13:4161 , piece_id=K4CTNQGHL5S2GOARW2F42REGZ4QVWA2RKUNULONYX3VPVHAGONUQ, seg_pos={0 0}, upload=533792]: Done uploading piece. err: <nil> Dec 31 00:33:27.347765488 (609h4m27.218848533s) [noise=true , address=45.91.116.12:4248 , piece_id=QXKIUKBDX2GSJ2EQPALFCLPPLNWBPL3GRAEHLPJNSWUDIMU7WHYA, seg_pos={0 0}, upload=533792]: Done uploading piece. err: <nil> Dec 31 00:33:27.789508217 (609h4m27.660591252s) [noise=true , address=145.79.227.44:4217 , piece_id=GW44B36EHWVCRLEM6BA5MXC3ITRNCCJATW2RXSYTNC7K7WRO2IDQ, seg_pos={0 0}, upload=533792]: Done uploading piece. err: <nil> Dec 31 00:33:27.821042021 (609h4m27.692125056s) [noise=true , address=145.79.227.49:4353 , piece_id=MVTAW6QPJY2LRNEGISHNDP7AC5ZIIUR5N5BBZRFO5I6VYJU7UR3Q, seg_pos={0 0}, upload=533792]: Done uploading piece. err: <nil> Dec 31 00:33:27.828648224 (609h4m27.699731259s) [noise=true , address=145.79.227.49:4370 , piece_id=52YCNJY7BLJ3N3HDWBZCM7PP2ZPHHBS4GV4QS4WM2AWSLYKB5BBA, seg_pos={0 0}, upload=533792]: Done uploading piece. err: <nil> Dec 31 00:33:27.840271497 (609h4m27.711354542s) [noise=true , address=145.79.227.30:4769 , piece_id=P2ANRKDKCOTOMCZZMDIH2YRVI366UOHWO3LXV7INEDB2GW35Y52A, seg_pos={0 0}, upload=533792]: Done uploading piece. err: <nil> Dec 31 00:33:27.844064569 (609h4m27.715147614s) [noise=true , address=145.79.227.34:4930 , piece_id=L3AVRBWPDFTZANZKO4TGITCA3WHBECKEQ2BGMLFQJYN4CR7FSADA, seg_pos={0 0}, upload=533792]: Done uploading piece. err: <nil> Dec 31 00:33:27.867248213 (609h4m27.738331248s) [noise=true , address=77.90.31.18:4835 , piece_id=GRMJTN6ZDDYITHJWX5UGIAGLCOW6AKADHTK6UCVGRB437TUTDFGA, seg_pos={0 0}, upload=533792]: Done uploading piece. err: <nil> Dec 31 00:33:27.867270586 (609h4m27.738353621s) [seg_pos={0 0}, upload=533792]: Segment reached optimal threshold of 14 pieces. Dec 31 00:33:27.867476859 (609h4m27.738559904s) [noise=true , address=64.49.13.155:4913 , piece_id=E4WJCMYJVURPHENWEWR46KOAUAOZU7FW7WLC53DAU44BE7R3CHMQ, seg_pos={0 0}, upload=533792]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oyYUnWDyFo1npaKpBxMG5K31oUzJnTpcU1XHFmKux3MCVUPZV): protocol: expected piece hash; context canceled; EOF Dec 31 00:33:27.867485555 (609h4m27.73856859s) [noise=true , address=64.49.13.155:4913 , piece_id=E4WJCMYJVURPHENWEWR46KOAUAOZU7FW7WLC53DAU44BE7R3CHMQ, seg_pos={0 0}, upload=533792]: Long tail cancellation due to optimal threshold Dec 31 00:33:27.867516204 (609h4m27.738599249s) [noise=true , address=145.79.227.25:4630 , piece_id=5BVJWWAW2QUQABWWZCD4EW54ZVH3K5RJNM5EXJJUGXI33O6CC24A, seg_pos={0 0}, upload=533792]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12aETKgTzKw8fyYqbeDA2JzfxZnWn12xUPYK1eRt6TvBPPdRJDp): protocol: expected piece hash; context canceled; EOF Dec 31 00:33:27.867526553 (609h4m27.738609588s) [noise=true , address=145.79.227.25:4630 , piece_id=5BVJWWAW2QUQABWWZCD4EW54ZVH3K5RJNM5EXJJUGXI33O6CC24A, seg_pos={0 0}, upload=533792]: Long tail cancellation due to optimal threshold Dec 31 00:33:27.867714432 (609h4m27.738797477s) [noise=true , address=145.79.227.4:4189 , piece_id=ZZF4DSIPQBWKBD45UR2J5P75U5VVMTVB54NEX5QKZV4K2WJZ43BQ, seg_pos={0 0}, upload=533792]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jH3757i2F3q18XSpoXznsysZPaXmEnqynpLPDupgRfxJrTgKv): protocol: write tcp 188.166.225.170:57528->145.79.227.4:4189: use of closed network connection; write tcp 188.166.225.170:57528->145.79.227.4:4189: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57528->145.79.227.4:4189: use of closed network connection Dec 31 00:33:27.867723689 (609h4m27.738806724s) [noise=true , address=145.79.227.4:4189 , piece_id=ZZF4DSIPQBWKBD45UR2J5P75U5VVMTVB54NEX5QKZV4K2WJZ43BQ, seg_pos={0 0}, upload=533792]: Long tail cancellation due to optimal threshold Dec 31 00:33:27.867744699 (609h4m27.738827734s) [noise=true , address=145.79.227.2:4128 , piece_id=DW37Q32FFAZ37KHMOII6XGCJVDO2XTKED5TLOKG65JULWTXJIQEA, seg_pos={0 0}, upload=533792]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1wABkAtAjoYkKyNoAiaVCxSvY6WgvHvAYFPCjpNojtpiYiGtDS): protocol: expected piece hash; context canceled; EOF Dec 31 00:33:27.867753616 (609h4m27.738836651s) [noise=true , address=145.79.227.2:4128 , piece_id=DW37Q32FFAZ37KHMOII6XGCJVDO2XTKED5TLOKG65JULWTXJIQEA, seg_pos={0 0}, upload=533792]: Long tail cancellation due to optimal threshold Dec 31 00:33:27.867777462 (609h4m27.738860507s) [seg_pos={0 0}, upload=533792]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 31 00:33:27.867853687 (609h4m27.738936722s) [upload=533792]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 00:33:27.867859989 (609h4m27.738943024s) [upload=533792]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 00:33:28.167335511 (609h4m28.038418546s) [upload=533792]: Done uploading segments... Dec 31 00:33:28.167355899 (609h4m28.038438934s) [upload=533792]: Upload finished. err: <nil> Duration: 15.802335062s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 5.05436023s trace: https://jaeger.dev.storj.io/trace/63d344eb9b98ed45 FAIL (2025-12-31 00:45:45 UTC) orbiter warning: operation: DeleteObject uplink: metaclient: context deadline exceeded Duration: 5.054406599s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 15.677859962s trace: https://jaeger.dev.storj.io/trace/67f6164cdd15ab81 FAIL (2025-12-31 00:45:17 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== 31 00:33:25.026367244 (609h4m24.897450289s) [noise=true , address=145.79.227.2:4128 , piece_id=DW37Q32FFAZ37KHMOII6XGCJVDO2XTKED5TLOKG65JULWTXJIQEA, seg_pos={0 0}, upload=533792]: Uploading piece... Dec 31 00:33:25.026878247 (609h4m24.897961292s) [noise=true , address=145.79.227.44:4217 , piece_id=GW44B36EHWVCRLEM6BA5MXC3ITRNCCJATW2RXSYTNC7K7WRO2IDQ, seg_pos={0 0}, upload=533792]: Uploading piece... Dec 31 00:33:25.027303136 (609h4m24.898386171s) [noise=true , address=45.91.116.13:4161 , piece_id=K4CTNQGHL5S2GOARW2F42REGZ4QVWA2RKUNULONYX3VPVHAGONUQ, seg_pos={0 0}, upload=533792]: Uploading piece... Dec 31 00:33:25.027901605 (609h4m24.89898465s) [noise=true , address=77.90.31.13:4632 , piece_id=5KUEGWEJV5NJE2EX3S5S2G7TS5TK23ZQXV3WPA4LLHO65QJRVW6Q, seg_pos={0 0}, upload=533792]: Uploading piece... Dec 31 00:33:25.028257373 (609h4m24.899340408s) [noise=true , address=145.79.227.30:4769 , piece_id=P2ANRKDKCOTOMCZZMDIH2YRVI366UOHWO3LXV7INEDB2GW35Y52A, seg_pos={0 0}, upload=533792]: Uploading piece... Dec 31 00:33:25.028623710 (609h4m24.899706745s) [noise=true , address=145.79.227.34:4930 , piece_id=L3AVRBWPDFTZANZKO4TGITCA3WHBECKEQ2BGMLFQJYN4CR7FSADA, seg_pos={0 0}, upload=533792]: Uploading piece... Dec 31 00:33:25.028952256 (609h4m24.900035291s) [noise=true , address=145.79.227.25:4630 , piece_id=5BVJWWAW2QUQABWWZCD4EW54ZVH3K5RJNM5EXJJUGXI33O6CC24A, seg_pos={0 0}, upload=533792]: Uploading piece... Dec 31 00:33:25.029314766 (609h4m24.900397801s) [noise=true , address=145.79.227.29:4971 , piece_id=EIY6YRG5ZRMEW7IWE5X25RCTGNCU3HQ27O7Q434OAZ4EI335HU4A, seg_pos={0 0}, upload=533792]: Uploading piece... Dec 31 00:33:25.029659533 (609h4m24.900742578s) [noise=true , address=145.79.227.6:4283 , piece_id=MXSG7TVTQBR2AFONVXYWQB4DBAQEEXW226FDC6SSVG5TTWGPAWFQ, seg_pos={0 0}, upload=533792]: Uploading piece... Dec 31 00:33:25.030576118 (609h4m24.901659153s) [noise=true , address=77.90.31.18:4835 , piece_id=GRMJTN6ZDDYITHJWX5UGIAGLCOW6AKADHTK6UCVGRB437TUTDFGA, seg_pos={0 0}, upload=533792]: Uploading piece... Dec 31 00:33:25.031413452 (609h4m24.902496487s) [noise=true , address=64.49.13.155:4913 , piece_id=E4WJCMYJVURPHENWEWR46KOAUAOZU7FW7WLC53DAU44BE7R3CHMQ, seg_pos={0 0}, upload=533792]: Uploading piece... Dec 31 00:33:25.031987204 (609h4m24.903070239s) [noise=true , address=145.79.227.49:4370 , piece_id=52YCNJY7BLJ3N3HDWBZCM7PP2ZPHHBS4GV4QS4WM2AWSLYKB5BBA, seg_pos={0 0}, upload=533792]: Uploading piece... Dec 31 00:33:25.032849947 (609h4m24.903932993s) [noise=true , address=145.79.227.4:4189 , piece_id=ZZF4DSIPQBWKBD45UR2J5P75U5VVMTVB54NEX5QKZV4K2WJZ43BQ, seg_pos={0 0}, upload=533792]: Uploading piece... Dec 31 00:33:25.034222320 (609h4m24.905305355s) [noise=true , address=145.79.227.49:4353 , piece_id=MVTAW6QPJY2LRNEGISHNDP7AC5ZIIUR5N5BBZRFO5I6VYJU7UR3Q, seg_pos={0 0}, upload=533792]: Uploading piece... Dec 31 00:33:25.035359376 (609h4m24.906442411s) [noise=true , address=145.79.227.35:4494 , piece_id=EY2OJLQIF4CH7DQWJWXSBJMQI37NAXSBLQP4ZPMU3IBRVLWJVYWA, seg_pos={0 0}, upload=533792]: Uploading piece... Dec 31 00:33:25.035852344 (609h4m24.906935379s) [noise=true , address=145.79.227.33:4873 , piece_id=2DSEGSC5UWL76NEXSE7WXOGSQNRNXALAINY52TLQHB57LTS2R6ZQ, seg_pos={0 0}, upload=533792]: Uploading piece... Dec 31 00:33:26.491429069 (609h4m26.362512104s) [upload=533792]: Next returned nil segment Dec 31 00:33:26.491441904 (609h4m26.362524939s) [upload=533792]: Waiting for error group managing segments... Dec 31 00:33:26.876786140 (609h4m26.747869175s) [noise=true , address=145.79.227.6:4283 , piece_id=MXSG7TVTQBR2AFONVXYWQB4DBAQEEXW226FDC6SSVG5TTWGPAWFQ, seg_pos={0 0}, upload=533792]: Done uploading piece. err: <nil> Dec 31 00:33:26.895923271 (609h4m26.767006316s) [noise=true , address=145.79.227.35:4494 , piece_id=EY2OJLQIF4CH7DQWJWXSBJMQI37NAXSBLQP4ZPMU3IBRVLWJVYWA, seg_pos={0 0}, upload=533792]: Done uploading piece. err: <nil> Dec 31 00:33:26.903106888 (609h4m26.774189923s) [noise=true , address=145.79.227.33:4873 , piece_id=2DSEGSC5UWL76NEXSE7WXOGSQNRNXALAINY52TLQHB57LTS2R6ZQ, seg_pos={0 0}, upload=533792]: Done uploading piece. err: <nil> Dec 31 00:33:26.912484285 (609h4m26.78356733s) [noise=true , address=145.79.227.29:4971 , piece_id=EIY6YRG5ZRMEW7IWE5X25RCTGNCU3HQ27O7Q434OAZ4EI335HU4A, seg_pos={0 0}, upload=533792]: Done uploading piece. err: <nil> Dec 31 00:33:27.004280888 (609h4m26.875363933s) [noise=true , address=145.79.227.21:4495 , piece_id=YWETKWGBGTY3LALK4DXWSKUTM2TFEYB4N5A3KNKW346V3F5XRN2Q, seg_pos={0 0}, upload=533792]: Done uploading piece. err: <nil> Dec 31 00:33:27.118759862 (609h4m26.989842897s) [noise=true , address=77.90.31.13:4632 , piece_id=5KUEGWEJV5NJE2EX3S5S2G7TS5TK23ZQXV3WPA4LLHO65QJRVW6Q, seg_pos={0 0}, upload=533792]: Done uploading piece. err: <nil> Dec 31 00:33:27.225324635 (609h4m27.09640767s) [noise=true , address=45.91.116.13:4161 , piece_id=K4CTNQGHL5S2GOARW2F42REGZ4QVWA2RKUNULONYX3VPVHAGONUQ, seg_pos={0 0}, upload=533792]: Done uploading piece. err: <nil> Dec 31 00:33:27.347765488 (609h4m27.218848533s) [noise=true , address=45.91.116.12:4248 , piece_id=QXKIUKBDX2GSJ2EQPALFCLPPLNWBPL3GRAEHLPJNSWUDIMU7WHYA, seg_pos={0 0}, upload=533792]: Done uploading piece. err: <nil> Dec 31 00:33:27.789508217 (609h4m27.660591252s) [noise=true , address=145.79.227.44:4217 , piece_id=GW44B36EHWVCRLEM6BA5MXC3ITRNCCJATW2RXSYTNC7K7WRO2IDQ, seg_pos={0 0}, upload=533792]: Done uploading piece. err: <nil> Dec 31 00:33:27.821042021 (609h4m27.692125056s) [noise=true , address=145.79.227.49:4353 , piece_id=MVTAW6QPJY2LRNEGISHNDP7AC5ZIIUR5N5BBZRFO5I6VYJU7UR3Q, seg_pos={0 0}, upload=533792]: Done uploading piece. err: <nil> Dec 31 00:33:27.828648224 (609h4m27.699731259s) [noise=true , address=145.79.227.49:4370 , piece_id=52YCNJY7BLJ3N3HDWBZCM7PP2ZPHHBS4GV4QS4WM2AWSLYKB5BBA, seg_pos={0 0}, upload=533792]: Done uploading piece. err: <nil> Dec 31 00:33:27.840271497 (609h4m27.711354542s) [noise=true , address=145.79.227.30:4769 , piece_id=P2ANRKDKCOTOMCZZMDIH2YRVI366UOHWO3LXV7INEDB2GW35Y52A, seg_pos={0 0}, upload=533792]: Done uploading piece. err: <nil> Dec 31 00:33:27.844064569 (609h4m27.715147614s) [noise=true , address=145.79.227.34:4930 , piece_id=L3AVRBWPDFTZANZKO4TGITCA3WHBECKEQ2BGMLFQJYN4CR7FSADA, seg_pos={0 0}, upload=533792]: Done uploading piece. err: <nil> Dec 31 00:33:27.867248213 (609h4m27.738331248s) [noise=true , address=77.90.31.18:4835 , piece_id=GRMJTN6ZDDYITHJWX5UGIAGLCOW6AKADHTK6UCVGRB437TUTDFGA, seg_pos={0 0}, upload=533792]: Done uploading piece. err: <nil> Dec 31 00:33:27.867270586 (609h4m27.738353621s) [seg_pos={0 0}, upload=533792]: Segment reached optimal threshold of 14 pieces. Dec 31 00:33:27.867476859 (609h4m27.738559904s) [noise=true , address=64.49.13.155:4913 , piece_id=E4WJCMYJVURPHENWEWR46KOAUAOZU7FW7WLC53DAU44BE7R3CHMQ, seg_pos={0 0}, upload=533792]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oyYUnWDyFo1npaKpBxMG5K31oUzJnTpcU1XHFmKux3MCVUPZV): protocol: expected piece hash; context canceled; EOF Dec 31 00:33:27.867485555 (609h4m27.73856859s) [noise=true , address=64.49.13.155:4913 , piece_id=E4WJCMYJVURPHENWEWR46KOAUAOZU7FW7WLC53DAU44BE7R3CHMQ, seg_pos={0 0}, upload=533792]: Long tail cancellation due to optimal threshold Dec 31 00:33:27.867516204 (609h4m27.738599249s) [noise=true , address=145.79.227.25:4630 , piece_id=5BVJWWAW2QUQABWWZCD4EW54ZVH3K5RJNM5EXJJUGXI33O6CC24A, seg_pos={0 0}, upload=533792]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12aETKgTzKw8fyYqbeDA2JzfxZnWn12xUPYK1eRt6TvBPPdRJDp): protocol: expected piece hash; context canceled; EOF Dec 31 00:33:27.867526553 (609h4m27.738609588s) [noise=true , address=145.79.227.25:4630 , piece_id=5BVJWWAW2QUQABWWZCD4EW54ZVH3K5RJNM5EXJJUGXI33O6CC24A, seg_pos={0 0}, upload=533792]: Long tail cancellation due to optimal threshold Dec 31 00:33:27.867714432 (609h4m27.738797477s) [noise=true , address=145.79.227.4:4189 , piece_id=ZZF4DSIPQBWKBD45UR2J5P75U5VVMTVB54NEX5QKZV4K2WJZ43BQ, seg_pos={0 0}, upload=533792]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jH3757i2F3q18XSpoXznsysZPaXmEnqynpLPDupgRfxJrTgKv): protocol: write tcp 188.166.225.170:57528->145.79.227.4:4189: use of closed network connection; write tcp 188.166.225.170:57528->145.79.227.4:4189: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57528->145.79.227.4:4189: use of closed network connection Dec 31 00:33:27.867723689 (609h4m27.738806724s) [noise=true , address=145.79.227.4:4189 , piece_id=ZZF4DSIPQBWKBD45UR2J5P75U5VVMTVB54NEX5QKZV4K2WJZ43BQ, seg_pos={0 0}, upload=533792]: Long tail cancellation due to optimal threshold Dec 31 00:33:27.867744699 (609h4m27.738827734s) [noise=true , address=145.79.227.2:4128 , piece_id=DW37Q32FFAZ37KHMOII6XGCJVDO2XTKED5TLOKG65JULWTXJIQEA, seg_pos={0 0}, upload=533792]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1wABkAtAjoYkKyNoAiaVCxSvY6WgvHvAYFPCjpNojtpiYiGtDS): protocol: expected piece hash; context canceled; EOF Dec 31 00:33:27.867753616 (609h4m27.738836651s) [noise=true , address=145.79.227.2:4128 , piece_id=DW37Q32FFAZ37KHMOII6XGCJVDO2XTKED5TLOKG65JULWTXJIQEA, seg_pos={0 0}, upload=533792]: Long tail cancellation due to optimal threshold Dec 31 00:33:27.867777462 (609h4m27.738860507s) [seg_pos={0 0}, upload=533792]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 31 00:33:27.867853687 (609h4m27.738936722s) [upload=533792]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 00:33:27.867859989 (609h4m27.738943024s) [upload=533792]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 00:33:28.167335511 (609h4m28.038418546s) [upload=533792]: Done uploading segments... Dec 31 00:33:28.167355899 (609h4m28.038438934s) [upload=533792]: Upload finished. err: <nil> Duration: 15.677886652s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [.......X...X...........X........X..X....XX.XXX....X.......XX] duration: 30.00245496s trace: https://jaeger.dev.storj.io/trace/6c07707c87e1fb63 FAIL (2025-12-31 00:45:14 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, Piece ID: GG4MQX6GQGCO54R7B2FQFKF3AVDCV6BGGU7GMDAVNYWJ23AESLGQ) context deadline exceeded; piecestore: (Node ID: 12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f, Piece ID: 7QVQHE6NXP7GGAHH5AYHXGL53OML2SVEYN42XXFUEZFMAI3GZDZA) context deadline exceeded; piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: CCLTLM57NIXBAAODKCOZ4YVX37CDV33A7A3WYR2PSSHEIKO2RQ2Q) context deadline exceeded; piecestore: (Node ID: 12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, Piece ID: QHK56YLQYVIN4LADYSB3YKE6TF3LTDXCEZOGH4KE4FVC2A2W42CA) context deadline exceeded ======LOGS====== Dec 31 00:44:45.443637561 (609h15m45.314720596s) [upload=533955]: Starting upload Dec 31 00:44:45.443709788 (609h15m45.314792833s) [upload=533955]: Done starting upload Dec 31 00:44:45.443796563 (609h15m45.314879608s) [upload=533955]: Uploading segments... Dec 31 00:44:45.443910320 (609h15m45.314993355s) [upload=533955]: Got next segment. Inline: false Dec 31 00:44:45.443944105 (609h15m45.31502714s) [upload=533955]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 00:44:45.443948483 (609h15m45.315031518s) [upload=533955]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 00:44:45.691936380 (609h15m45.563019415s) [seg_pos={0 0}, upload=533955]: Begin upload segment... Dec 31 00:44:45.691985453 (609h15m45.563068488s) [seg_pos={0 0}, upload=533955]: Done begin upload segment. Dec 31 00:44:45.692065505 (609h15m45.56314854s) [noise=true , address=145.79.227.44:4194 , piece_id=QYEHIRT3MVMVKM6YWOW7DL7QPY5UPXJCRVEOLMIS4N4JGYOXCG7A, seg_pos={0 0}, upload=533955]: Uploading piece... Dec 31 00:44:45.692086035 (609h15m45.56316907s) [noise=true , address=145.79.227.32:4834 , piece_id=QX55FO4R3LZHFC2QBFRMZLAMGXGDSUH2TH4FD7BX5D3RR6BSXSQQ, seg_pos={0 0}, upload=533955]: Uploading piece... Dec 31 00:44:45.692196725 (609h15m45.56327976s) [noise=true , address=45.91.116.6:4975 , piece_id=CCLTLM57NIXBAAODKCOZ4YVX37CDV33A7A3WYR2PSSHEIKO2RQ2Q, seg_pos={0 0}, upload=533955]: Uploading piece... Dec 31 00:44:45.692602398 (609h15m45.563685433s) [noise=true , address=77.90.31.11:4555 , piece_id=D7HD3K56K6GLAKLIA4AHV4FBIOICZVKO467WYG3VI6YD7WBAWU6Q, seg_pos={0 0}, upload=533955]: Uploading piece... Dec 31 00:44:45.692612487 (609h15m45.563695522s) [noise=true , address=45.91.116.4:4940 , piece_id=QVIN7MZ5TO2DI257G4RXFIRASHBCUT7J327MDWMOLRSHGO4OTP6Q, seg_pos={0 0}, upload=533955]: Uploading piece... Dec 31 00:44:45.692917979 (609h15m45.564001014s) [noise=true , address=145.79.227.21:4475 , piece_id=DDI4OGAJOIDAY5QWK33H5YA6VWSKXET2MSFDQ7VLC6UQ7WDGSRLQ, seg_pos={0 0}, upload=533955]: Uploading piece... Dec 31 00:44:45.693173665 (609h15m45.564256751s) [noise=true , address=145.79.227.19:4078 , piece_id=PTQY2TGFMUEBJ2PGB5WWW6ICT453L7CXY322TMS6655Z5UJ6GBVA, seg_pos={0 0}, upload=533955]: Uploading piece... Dec 31 00:44:45.693253016 (609h15m45.564336061s) [noise=true , address=145.79.227.2:4115 , piece_id=XBWADCTRARLWVQWHZ3IVMXDN2FYEE34WWZ5YFQE24BM7VGTRPNIA, seg_pos={0 0}, upload=533955]: Uploading piece... Dec 31 00:44:45.693476122 (609h15m45.564559157s) [noise=true , address=45.91.116.12:4227 , piece_id=QHK56YLQYVIN4LADYSB3YKE6TF3LTDXCEZOGH4KE4FVC2A2W42CA, seg_pos={0 0}, upload=533955]: Uploading piece... Dec 31 00:44:45.693772385 (609h15m45.56485542s) [noise=true , address=45.91.116.7:4009 , piece_id=BTIV67EFKX5OFXBIIGGS2GDXMUHR3EHG4S6ZMI3LQFSPLV4PVR5A, seg_pos={0 0}, upload=533955]: Uploading piece... Dec 31 00:44:45.694100911 (609h15m45.565183946s) [noise=true , address=145.79.227.22:4511 , piece_id=C2R77ZH5L72IWZEOZSYQOVIH4MD7Z2MTXVK7EF6IWH5DEUPOLW7Q, seg_pos={0 0}, upload=533955]: Uploading piece... Dec 31 00:44:45.694395382 (609h15m45.565478417s) [noise=true , address=145.79.227.13:4802 , piece_id=TXFKE7MTYB5NNC762QORY3UDB5Q42HJ2J4JEKQZIXIUFQYQBFZ5Q, seg_pos={0 0}, upload=533955]: Uploading piece... Dec 31 00:44:45.694842984 (609h15m45.565926019s) [noise=true , address=45.91.116.5:4298 , piece_id=GG4MQX6GQGCO54R7B2FQFKF3AVDCV6BGGU7GMDAVNYWJ23AESLGQ, seg_pos={0 0}, upload=533955]: Uploading piece... Dec 31 00:44:45.695220373 (609h15m45.566303408s) [noise=true , address=145.79.227.23:4546 , piece_id=5RMH6VUR7LQBHGNYAE7R4O4GF3GGHN5ZVL7Q6C6V3YJ4HGFUS3YA, seg_pos={0 0}, upload=533955]: Uploading piece... Dec 31 00:44:45.696046176 (609h15m45.567129211s) [noise=true , address=145.79.227.5:4222 , piece_id=I55PDNWS2YXLSQGEW23FNSXQQQDANQBHAR2T2OH2V4CCA2P7OHFQ, seg_pos={0 0}, upload=533955]: Uploading piece... Dec 31 00:44:45.697349988 (609h15m45.568433023s) [noise=true , address=77.90.31.13:4629 , piece_id=7QVQHE6NXP7GGAHH5AYHXGL53OML2SVEYN42XXFUEZFMAI3GZDZA, seg_pos={0 0}, upload=533955]: Uploading piece... Dec 31 00:44:47.389082715 (609h15m47.26016575s) [upload=533955]: Next returned nil segment Dec 31 00:44:47.389100959 (609h15m47.260183994s) [upload=533955]: Waiting for error group managing segments... Dec 31 00:44:47.679637866 (609h15m47.550720911s) [noise=true , address=145.79.227.22:4511 , piece_id=C2R77ZH5L72IWZEOZSYQOVIH4MD7Z2MTXVK7EF6IWH5DEUPOLW7Q, seg_pos={0 0}, upload=533955]: Done uploading piece. err: <nil> Dec 31 00:44:47.681072347 (609h15m47.552155382s) [noise=true , address=145.79.227.23:4546 , piece_id=5RMH6VUR7LQBHGNYAE7R4O4GF3GGHN5ZVL7Q6C6V3YJ4HGFUS3YA, seg_pos={0 0}, upload=533955]: Done uploading piece. err: <nil> Dec 31 00:44:47.681636803 (609h15m47.552719838s) [noise=true , address=145.79.227.21:4475 , piece_id=DDI4OGAJOIDAY5QWK33H5YA6VWSKXET2MSFDQ7VLC6UQ7WDGSRLQ, seg_pos={0 0}, upload=533955]: Done uploading piece. err: <nil> Dec 31 00:44:47.682153145 (609h15m47.5532362s) [noise=true , address=145.79.227.2:4115 , piece_id=XBWADCTRARLWVQWHZ3IVMXDN2FYEE34WWZ5YFQE24BM7VGTRPNIA, seg_pos={0 0}, upload=533955]: Done uploading piece. err: <nil> Dec 31 00:44:47.684310152 (609h15m47.555393207s) [noise=true , address=145.79.227.32:4834 , piece_id=QX55FO4R3LZHFC2QBFRMZLAMGXGDSUH2TH4FD7BX5D3RR6BSXSQQ, seg_pos={0 0}, upload=533955]: Done uploading piece. err: <nil> Dec 31 00:44:47.693448863 (609h15m47.564531909s) [noise=true , address=145.79.227.13:4802 , piece_id=TXFKE7MTYB5NNC762QORY3UDB5Q42HJ2J4JEKQZIXIUFQYQBFZ5Q, seg_pos={0 0}, upload=533955]: Done uploading piece. err: <nil> Dec 31 00:44:47.697022899 (609h15m47.568105944s) [noise=true , address=145.79.227.5:4222 , piece_id=I55PDNWS2YXLSQGEW23FNSXQQQDANQBHAR2T2OH2V4CCA2P7OHFQ, seg_pos={0 0}, upload=533955]: Done uploading piece. err: <nil> Dec 31 00:44:47.761759876 (609h15m47.632842911s) [noise=true , address=145.79.227.19:4078 , piece_id=PTQY2TGFMUEBJ2PGB5WWW6ICT453L7CXY322TMS6655Z5UJ6GBVA, seg_pos={0 0}, upload=533955]: Done uploading piece. err: <nil> Dec 31 00:44:48.007827617 (609h15m47.878910662s) [noise=true , address=45.91.116.4:4940 , piece_id=QVIN7MZ5TO2DI257G4RXFIRASHBCUT7J327MDWMOLRSHGO4OTP6Q, seg_pos={0 0}, upload=533955]: Done uploading piece. err: <nil> Dec 31 00:44:48.012739720 (609h15m47.883822765s) [noise=true , address=45.91.116.6:4975 , piece_id=CCLTLM57NIXBAAODKCOZ4YVX37CDV33A7A3WYR2PSSHEIKO2RQ2Q, seg_pos={0 0}, upload=533955]: Done uploading piece. err: <nil> Dec 31 00:44:48.026107714 (609h15m47.897190749s) [noise=true , address=45.91.116.5:4298 , piece_id=GG4MQX6GQGCO54R7B2FQFKF3AVDCV6BGGU7GMDAVNYWJ23AESLGQ, seg_pos={0 0}, upload=533955]: Done uploading piece. err: <nil> Dec 31 00:44:48.186776747 (609h15m48.057859792s) [noise=true , address=77.90.31.13:4629 , piece_id=7QVQHE6NXP7GGAHH5AYHXGL53OML2SVEYN42XXFUEZFMAI3GZDZA, seg_pos={0 0}, upload=533955]: Done uploading piece. err: <nil> Dec 31 00:44:48.360970714 (609h15m48.232053749s) [noise=true , address=45.91.116.12:4227 , piece_id=QHK56YLQYVIN4LADYSB3YKE6TF3LTDXCEZOGH4KE4FVC2A2W42CA, seg_pos={0 0}, upload=533955]: Done uploading piece. err: <nil> Dec 31 00:44:48.360990682 (609h15m48.232073727s) [seg_pos={0 0}, upload=533955]: Segment reached optimal threshold of 13 pieces. Dec 31 00:44:48.361159894 (609h15m48.232242939s) [noise=true , address=45.91.116.7:4009 , piece_id=BTIV67EFKX5OFXBIIGGS2GDXMUHR3EHG4S6ZMI3LQFSPLV4PVR5A, seg_pos={0 0}, upload=533955]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: write tcp 188.166.225.170:52828->45.91.116.7:4009: use of closed network connection; write tcp 188.166.225.170:52828->45.91.116.7:4009: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52828->45.91.116.7:4009: use of closed network connection Dec 31 00:44:48.361165144 (609h15m48.232248189s) [noise=true , address=45.91.116.7:4009 , piece_id=BTIV67EFKX5OFXBIIGGS2GDXMUHR3EHG4S6ZMI3LQFSPLV4PVR5A, seg_pos={0 0}, upload=533955]: Long tail cancellation due to optimal threshold Dec 31 00:44:48.361331891 (609h15m48.232414946s) [noise=true , address=145.79.227.44:4194 , piece_id=QYEHIRT3MVMVKM6YWOW7DL7QPY5UPXJCRVEOLMIS4N4JGYOXCG7A, seg_pos={0 0}, upload=533955]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh): protocol: expected piece hash; context canceled; EOF Dec 31 00:44:48.361343514 (609h15m48.232426559s) [noise=true , address=145.79.227.44:4194 , piece_id=QYEHIRT3MVMVKM6YWOW7DL7QPY5UPXJCRVEOLMIS4N4JGYOXCG7A, seg_pos={0 0}, upload=533955]: Long tail cancellation due to optimal threshold Dec 31 00:44:48.361388970 (609h15m48.232472005s) [noise=true , address=77.90.31.11:4555 , piece_id=D7HD3K56K6GLAKLIA4AHV4FBIOICZVKO467WYG3VI6YD7WBAWU6Q, seg_pos={0 0}, upload=533955]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ): protocol: expected piece hash; context canceled; EOF Dec 31 00:44:48.361395132 (609h15m48.232478167s) [noise=true , address=77.90.31.11:4555 , piece_id=D7HD3K56K6GLAKLIA4AHV4FBIOICZVKO467WYG3VI6YD7WBAWU6Q, seg_pos={0 0}, upload=533955]: Long tail cancellation due to optimal threshold Dec 31 00:44:48.361412014 (609h15m48.232495049s) [seg_pos={0 0}, upload=533955]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 00:44:48.361474594 (609h15m48.232557629s) [upload=533955]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 00:44:48.361479443 (609h15m48.232562478s) [upload=533955]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 00:44:48.655948215 (609h15m48.52703125s) [upload=533955]: Done uploading segments... Dec 31 00:44:48.655991958 (609h15m48.527074993s) [upload=533955]: Upload finished. err: <nil> Duration: 30.002464088s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.....................................................X......] duration: 3.588885494s trace: https://jaeger.dev.storj.io/trace/2c1d476356ae4e72 FAIL (2025-12-31 00:45:13 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.588910611s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 4.815623527s trace: https://jaeger.dev.storj.io/trace/63d344eb9b98ed45 FAIL (2025-12-31 00:45:11 UTC) orbiter warning: operation: DeleteObject uplink: metaclient: context deadline exceeded Duration: 4.815653815s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [........X...X...........X........X..X....XX.XXX....X.......X] duration: 30.003613232s trace: https://jaeger.dev.storj.io/trace/66934e5f43a7e33e FAIL (2025-12-31 00:44:44 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, Piece ID: WQBM4CNK7UVPWCMNRRZJHD6ZMWUU4MTMM4PXVZ5XCHY4THXRDMCQ) context deadline exceeded; piecestore: (Node ID: 1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, Piece ID: RQTX3ZMCCJL5AD6PGBNHCZM7V2L5MBYGME3NQ6FY5LQAOIN776NA) context deadline exceeded ======LOGS====== ents... Dec 31 00:44:15.209035721 (609h15m15.080118766s) [upload=533948]: Got next segment. Inline: false Dec 31 00:44:15.209083672 (609h15m15.080166707s) [upload=533948]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 00:44:15.209090234 (609h15m15.080173279s) [upload=533948]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 00:44:15.456393157 (609h15m15.327476202s) [seg_pos={0 0}, upload=533948]: Begin upload segment... Dec 31 00:44:15.456461557 (609h15m15.327544623s) [seg_pos={0 0}, upload=533948]: Done begin upload segment. Dec 31 00:44:15.456547712 (609h15m15.327630747s) [noise=true , address=145.79.227.50:4373 , piece_id=DKGGB5SKJDAAGGXAMFT4B7W7YDNKRZ6C3YQCM732A54FEGUFW6ZQ, seg_pos={0 0}, upload=533948]: Uploading piece... Dec 31 00:44:15.456589802 (609h15m15.327672837s) [noise=true , address=77.90.31.15:4700 , piece_id=WQBM4CNK7UVPWCMNRRZJHD6ZMWUU4MTMM4PXVZ5XCHY4THXRDMCQ, seg_pos={0 0}, upload=533948]: Uploading piece... Dec 31 00:44:15.457125652 (609h15m15.328208697s) [noise=true , address=77.90.31.11:4555 , piece_id=EXJP6BUKLCZ635HGWV623FZ5PPFMR6Y4KCDJJX3WFNC5PMMRGNHA, seg_pos={0 0}, upload=533948]: Uploading piece... Dec 31 00:44:15.457499484 (609h15m15.328582529s) [noise=true , address=145.79.227.14:4838 , piece_id=DMNHXHNDUP3K4J2PML34QKCLFVPJPHK6PDPLJ2NKRKEHIDAHYOJQ, seg_pos={0 0}, upload=533948]: Uploading piece... Dec 31 00:44:15.457834061 (609h15m15.328917097s) [noise=true , address=77.90.31.14:4664 , piece_id=E4MQG7M2CDWDU2HXS2EFPBHXUQFLWY42L7H5G4ZJYRB4B2JRRSQQ, seg_pos={0 0}, upload=533948]: Uploading piece... Dec 31 00:44:15.459403119 (609h15m15.330486164s) [noise=true , address=45.91.116.9:4261 , piece_id=QMQMGM2RPXJRTLLNIARNOB4AUEJDAU4NFZSFNFQEEYO7T5S5JXCA, seg_pos={0 0}, upload=533948]: Uploading piece... Dec 31 00:44:15.459871370 (609h15m15.330954415s) [noise=true , address=77.90.31.19:4841 , piece_id=RQTX3ZMCCJL5AD6PGBNHCZM7V2L5MBYGME3NQ6FY5LQAOIN776NA, seg_pos={0 0}, upload=533948]: Uploading piece... Dec 31 00:44:15.460276973 (609h15m15.331360018s) [noise=true , address=145.79.227.2:4115 , piece_id=33MU4WSYHZ7PROJ27CUGD7DG72PFKAB6EABA4XGOWTGEJ7LJE5LQ, seg_pos={0 0}, upload=533948]: Uploading piece... Dec 31 00:44:15.460608935 (609h15m15.33169197s) [noise=true , address=77.90.31.12:4590 , piece_id=DHUPP44SR3UGY436M3NVB6GPYGFXBQ3JD2FWYMP63FNZHWWDYLEQ, seg_pos={0 0}, upload=533948]: Uploading piece... Dec 31 00:44:15.460951126 (609h15m15.332034161s) [noise=true , address=145.79.227.43:4159 , piece_id=F5GGJ72OPO4UUVAF57CKNC7RF3Q3EZHW4MII24GBFLA5YB6N7HDQ, seg_pos={0 0}, upload=533948]: Uploading piece... Dec 31 00:44:15.461274562 (609h15m15.332357607s) [noise=true , address=45.91.116.7:4011 , piece_id=GUJ2QFPCBI2AVHVXQDGCBMJ7ONC7SHR53YOGL66Q7GGZCPCP5CAA, seg_pos={0 0}, upload=533948]: Uploading piece... Dec 31 00:44:15.461674394 (609h15m15.332757429s) [noise=true , address=145.79.227.26:4653 , piece_id=537XCKRZCWYXSV4NDDZFCRORHYGS36LFOIMYKD5FWUL3VEKQMFQQ, seg_pos={0 0}, upload=533948]: Uploading piece... Dec 31 00:44:15.466514290 (609h15m15.337597335s) [noise=true , address=77.90.31.21:4877 , piece_id=QFJZRRQVGADJPU3WAM4XFHKYJZQQ2K6OOTXWNVKFSPVXZTA52TJQ, seg_pos={0 0}, upload=533948]: Uploading piece... Dec 31 00:44:15.468909680 (609h15m15.339992725s) [noise=true , address=45.91.116.13:4153 , piece_id=LNNMU54MYFMPSEFQDYQGSZPXLILDHV47SXMW7ZQFEB342CBEQZ3Q, seg_pos={0 0}, upload=533948]: Uploading piece... Dec 31 00:44:15.471110661 (609h15m15.342193706s) [noise=true , address=145.79.227.11:4729 , piece_id=DHON27UQXTFWQP6UZZXO7FUHQUD36SVJLLG2EEVHZHUTKZYQRBJA, seg_pos={0 0}, upload=533948]: Uploading piece... Dec 31 00:44:15.472117709 (609h15m15.343200744s) [noise=true , address=145.79.227.13:4803 , piece_id=MC7KE2UW34UTNVRY77C4ZB37QJYX5ZWSQ46467NFDNXKSRYMIYQQ, seg_pos={0 0}, upload=533948]: Uploading piece... Dec 31 00:44:17.167930614 (609h15m17.039013659s) [upload=533948]: Next returned nil segment Dec 31 00:44:17.167947497 (609h15m17.039030532s) [upload=533948]: Waiting for error group managing segments... Dec 31 00:44:17.442232788 (609h15m17.313315823s) [noise=true , address=145.79.227.2:4115 , piece_id=33MU4WSYHZ7PROJ27CUGD7DG72PFKAB6EABA4XGOWTGEJ7LJE5LQ, seg_pos={0 0}, upload=533948]: Done uploading piece. err: <nil> Dec 31 00:44:17.451705725 (609h15m17.32278876s) [noise=true , address=145.79.227.11:4729 , piece_id=DHON27UQXTFWQP6UZZXO7FUHQUD36SVJLLG2EEVHZHUTKZYQRBJA, seg_pos={0 0}, upload=533948]: Done uploading piece. err: <nil> Dec 31 00:44:17.475574505 (609h15m17.34665755s) [noise=true , address=145.79.227.14:4838 , piece_id=DMNHXHNDUP3K4J2PML34QKCLFVPJPHK6PDPLJ2NKRKEHIDAHYOJQ, seg_pos={0 0}, upload=533948]: Done uploading piece. err: <nil> Dec 31 00:44:17.547890693 (609h15m17.418973728s) [noise=true , address=145.79.227.26:4653 , piece_id=537XCKRZCWYXSV4NDDZFCRORHYGS36LFOIMYKD5FWUL3VEKQMFQQ, seg_pos={0 0}, upload=533948]: Done uploading piece. err: <nil> Dec 31 00:44:17.613582819 (609h15m17.484665854s) [noise=true , address=77.90.31.11:4555 , piece_id=EXJP6BUKLCZ635HGWV623FZ5PPFMR6Y4KCDJJX3WFNC5PMMRGNHA, seg_pos={0 0}, upload=533948]: Done uploading piece. err: <nil> Dec 31 00:44:17.715206481 (609h15m17.586289527s) [noise=true , address=77.90.31.15:4700 , piece_id=WQBM4CNK7UVPWCMNRRZJHD6ZMWUU4MTMM4PXVZ5XCHY4THXRDMCQ, seg_pos={0 0}, upload=533948]: Done uploading piece. err: <nil> Dec 31 00:44:17.717390681 (609h15m17.588473716s) [noise=true , address=77.90.31.14:4664 , piece_id=E4MQG7M2CDWDU2HXS2EFPBHXUQFLWY42L7H5G4ZJYRB4B2JRRSQQ, seg_pos={0 0}, upload=533948]: Done uploading piece. err: <nil> Dec 31 00:44:17.731556224 (609h15m17.602639259s) [noise=true , address=77.90.31.21:4877 , piece_id=QFJZRRQVGADJPU3WAM4XFHKYJZQQ2K6OOTXWNVKFSPVXZTA52TJQ, seg_pos={0 0}, upload=533948]: Done uploading piece. err: <nil> Dec 31 00:44:17.747045898 (609h15m17.618128933s) [noise=true , address=145.79.227.13:4803 , piece_id=MC7KE2UW34UTNVRY77C4ZB37QJYX5ZWSQ46467NFDNXKSRYMIYQQ, seg_pos={0 0}, upload=533948]: Done uploading piece. err: <nil> Dec 31 00:44:17.847326002 (609h15m17.718409038s) [noise=true , address=45.91.116.9:4261 , piece_id=QMQMGM2RPXJRTLLNIARNOB4AUEJDAU4NFZSFNFQEEYO7T5S5JXCA, seg_pos={0 0}, upload=533948]: Done uploading piece. err: <nil> Dec 31 00:44:17.876864908 (609h15m17.747947953s) [noise=true , address=45.91.116.7:4011 , piece_id=GUJ2QFPCBI2AVHVXQDGCBMJ7ONC7SHR53YOGL66Q7GGZCPCP5CAA, seg_pos={0 0}, upload=533948]: Done uploading piece. err: <nil> Dec 31 00:44:18.011972087 (609h15m17.883055132s) [noise=true , address=45.91.116.13:4153 , piece_id=LNNMU54MYFMPSEFQDYQGSZPXLILDHV47SXMW7ZQFEB342CBEQZ3Q, seg_pos={0 0}, upload=533948]: Done uploading piece. err: <nil> Dec 31 00:44:18.370921187 (609h15m18.242004222s) [noise=true , address=77.90.31.19:4841 , piece_id=RQTX3ZMCCJL5AD6PGBNHCZM7V2L5MBYGME3NQ6FY5LQAOIN776NA, seg_pos={0 0}, upload=533948]: Done uploading piece. err: <nil> Dec 31 00:44:18.370942257 (609h15m18.242025303s) [seg_pos={0 0}, upload=533948]: Segment reached optimal threshold of 13 pieces. Dec 31 00:44:18.371093445 (609h15m18.24217648s) [noise=true , address=145.79.227.43:4159 , piece_id=F5GGJ72OPO4UUVAF57CKNC7RF3Q3EZHW4MII24GBFLA5YB6N7HDQ, seg_pos={0 0}, upload=533948]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ULUPKjJwZocjV1R6Dy7vmWSrZ4hKx3YAHLAihgtMKf8Uw2UgA): protocol: expected piece hash; context canceled; EOF Dec 31 00:44:18.371120707 (609h15m18.242203742s) [noise=true , address=145.79.227.43:4159 , piece_id=F5GGJ72OPO4UUVAF57CKNC7RF3Q3EZHW4MII24GBFLA5YB6N7HDQ, seg_pos={0 0}, upload=533948]: Long tail cancellation due to optimal threshold Dec 31 00:44:18.371266865 (609h15m18.2423499s) [noise=true , address=145.79.227.50:4373 , piece_id=DKGGB5SKJDAAGGXAMFT4B7W7YDNKRZ6C3YQCM732A54FEGUFW6ZQ, seg_pos={0 0}, upload=533948]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AQGSkJuNJDxoeqhChUenWxHwF5yK41UESCF63FQz3XvabXXfz): protocol: write tcp 188.166.225.170:45188->145.79.227.50:4373: use of closed network connection; write tcp 188.166.225.170:45188->145.79.227.50:4373: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45188->145.79.227.50:4373: use of closed network connection Dec 31 00:44:18.371282596 (609h15m18.242365641s) [noise=true , address=145.79.227.50:4373 , piece_id=DKGGB5SKJDAAGGXAMFT4B7W7YDNKRZ6C3YQCM732A54FEGUFW6ZQ, seg_pos={0 0}, upload=533948]: Long tail cancellation due to optimal threshold Dec 31 00:44:18.371314536 (609h15m18.242397571s) [noise=true , address=77.90.31.12:4590 , piece_id=DHUPP44SR3UGY436M3NVB6GPYGFXBQ3JD2FWYMP63FNZHWWDYLEQ, seg_pos={0 0}, upload=533948]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw): protocol: write tcp 188.166.225.170:54546->77.90.31.12:4590: use of closed network connection; write tcp 188.166.225.170:54546->77.90.31.12:4590: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54546->77.90.31.12:4590: use of closed network connection Dec 31 00:44:18.371320849 (609h15m18.242403884s) [noise=true , address=77.90.31.12:4590 , piece_id=DHUPP44SR3UGY436M3NVB6GPYGFXBQ3JD2FWYMP63FNZHWWDYLEQ, seg_pos={0 0}, upload=533948]: Long tail cancellation due to optimal threshold Dec 31 00:44:18.371395060 (609h15m18.242478095s) [seg_pos={0 0}, upload=533948]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 00:44:18.371475112 (609h15m18.242558147s) [upload=533948]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 00:44:18.371479941 (609h15m18.242562976s) [upload=533948]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 00:44:18.660801452 (609h15m18.531884497s) [upload=533948]: Done uploading segments... Dec 31 00:44:18.660821851 (609h15m18.531904886s) [upload=533948]: Upload finished. err: <nil> Duration: 30.003629453s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [......................................................X.....] duration: 3.637090407s trace: https://jaeger.dev.storj.io/trace/2c1d476356ae4e72 FAIL (2025-12-31 00:44:39 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.637121155s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 30.79061776s trace: https://jaeger.dev.storj.io/trace/67f6164cdd15ab81 FAIL (2025-12-31 00:44:38 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== 31 00:33:25.026367244 (609h4m24.897450289s) [noise=true , address=145.79.227.2:4128 , piece_id=DW37Q32FFAZ37KHMOII6XGCJVDO2XTKED5TLOKG65JULWTXJIQEA, seg_pos={0 0}, upload=533792]: Uploading piece... Dec 31 00:33:25.026878247 (609h4m24.897961292s) [noise=true , address=145.79.227.44:4217 , piece_id=GW44B36EHWVCRLEM6BA5MXC3ITRNCCJATW2RXSYTNC7K7WRO2IDQ, seg_pos={0 0}, upload=533792]: Uploading piece... Dec 31 00:33:25.027303136 (609h4m24.898386171s) [noise=true , address=45.91.116.13:4161 , piece_id=K4CTNQGHL5S2GOARW2F42REGZ4QVWA2RKUNULONYX3VPVHAGONUQ, seg_pos={0 0}, upload=533792]: Uploading piece... Dec 31 00:33:25.027901605 (609h4m24.89898465s) [noise=true , address=77.90.31.13:4632 , piece_id=5KUEGWEJV5NJE2EX3S5S2G7TS5TK23ZQXV3WPA4LLHO65QJRVW6Q, seg_pos={0 0}, upload=533792]: Uploading piece... Dec 31 00:33:25.028257373 (609h4m24.899340408s) [noise=true , address=145.79.227.30:4769 , piece_id=P2ANRKDKCOTOMCZZMDIH2YRVI366UOHWO3LXV7INEDB2GW35Y52A, seg_pos={0 0}, upload=533792]: Uploading piece... Dec 31 00:33:25.028623710 (609h4m24.899706745s) [noise=true , address=145.79.227.34:4930 , piece_id=L3AVRBWPDFTZANZKO4TGITCA3WHBECKEQ2BGMLFQJYN4CR7FSADA, seg_pos={0 0}, upload=533792]: Uploading piece... Dec 31 00:33:25.028952256 (609h4m24.900035291s) [noise=true , address=145.79.227.25:4630 , piece_id=5BVJWWAW2QUQABWWZCD4EW54ZVH3K5RJNM5EXJJUGXI33O6CC24A, seg_pos={0 0}, upload=533792]: Uploading piece... Dec 31 00:33:25.029314766 (609h4m24.900397801s) [noise=true , address=145.79.227.29:4971 , piece_id=EIY6YRG5ZRMEW7IWE5X25RCTGNCU3HQ27O7Q434OAZ4EI335HU4A, seg_pos={0 0}, upload=533792]: Uploading piece... Dec 31 00:33:25.029659533 (609h4m24.900742578s) [noise=true , address=145.79.227.6:4283 , piece_id=MXSG7TVTQBR2AFONVXYWQB4DBAQEEXW226FDC6SSVG5TTWGPAWFQ, seg_pos={0 0}, upload=533792]: Uploading piece... Dec 31 00:33:25.030576118 (609h4m24.901659153s) [noise=true , address=77.90.31.18:4835 , piece_id=GRMJTN6ZDDYITHJWX5UGIAGLCOW6AKADHTK6UCVGRB437TUTDFGA, seg_pos={0 0}, upload=533792]: Uploading piece... Dec 31 00:33:25.031413452 (609h4m24.902496487s) [noise=true , address=64.49.13.155:4913 , piece_id=E4WJCMYJVURPHENWEWR46KOAUAOZU7FW7WLC53DAU44BE7R3CHMQ, seg_pos={0 0}, upload=533792]: Uploading piece... Dec 31 00:33:25.031987204 (609h4m24.903070239s) [noise=true , address=145.79.227.49:4370 , piece_id=52YCNJY7BLJ3N3HDWBZCM7PP2ZPHHBS4GV4QS4WM2AWSLYKB5BBA, seg_pos={0 0}, upload=533792]: Uploading piece... Dec 31 00:33:25.032849947 (609h4m24.903932993s) [noise=true , address=145.79.227.4:4189 , piece_id=ZZF4DSIPQBWKBD45UR2J5P75U5VVMTVB54NEX5QKZV4K2WJZ43BQ, seg_pos={0 0}, upload=533792]: Uploading piece... Dec 31 00:33:25.034222320 (609h4m24.905305355s) [noise=true , address=145.79.227.49:4353 , piece_id=MVTAW6QPJY2LRNEGISHNDP7AC5ZIIUR5N5BBZRFO5I6VYJU7UR3Q, seg_pos={0 0}, upload=533792]: Uploading piece... Dec 31 00:33:25.035359376 (609h4m24.906442411s) [noise=true , address=145.79.227.35:4494 , piece_id=EY2OJLQIF4CH7DQWJWXSBJMQI37NAXSBLQP4ZPMU3IBRVLWJVYWA, seg_pos={0 0}, upload=533792]: Uploading piece... Dec 31 00:33:25.035852344 (609h4m24.906935379s) [noise=true , address=145.79.227.33:4873 , piece_id=2DSEGSC5UWL76NEXSE7WXOGSQNRNXALAINY52TLQHB57LTS2R6ZQ, seg_pos={0 0}, upload=533792]: Uploading piece... Dec 31 00:33:26.491429069 (609h4m26.362512104s) [upload=533792]: Next returned nil segment Dec 31 00:33:26.491441904 (609h4m26.362524939s) [upload=533792]: Waiting for error group managing segments... Dec 31 00:33:26.876786140 (609h4m26.747869175s) [noise=true , address=145.79.227.6:4283 , piece_id=MXSG7TVTQBR2AFONVXYWQB4DBAQEEXW226FDC6SSVG5TTWGPAWFQ, seg_pos={0 0}, upload=533792]: Done uploading piece. err: <nil> Dec 31 00:33:26.895923271 (609h4m26.767006316s) [noise=true , address=145.79.227.35:4494 , piece_id=EY2OJLQIF4CH7DQWJWXSBJMQI37NAXSBLQP4ZPMU3IBRVLWJVYWA, seg_pos={0 0}, upload=533792]: Done uploading piece. err: <nil> Dec 31 00:33:26.903106888 (609h4m26.774189923s) [noise=true , address=145.79.227.33:4873 , piece_id=2DSEGSC5UWL76NEXSE7WXOGSQNRNXALAINY52TLQHB57LTS2R6ZQ, seg_pos={0 0}, upload=533792]: Done uploading piece. err: <nil> Dec 31 00:33:26.912484285 (609h4m26.78356733s) [noise=true , address=145.79.227.29:4971 , piece_id=EIY6YRG5ZRMEW7IWE5X25RCTGNCU3HQ27O7Q434OAZ4EI335HU4A, seg_pos={0 0}, upload=533792]: Done uploading piece. err: <nil> Dec 31 00:33:27.004280888 (609h4m26.875363933s) [noise=true , address=145.79.227.21:4495 , piece_id=YWETKWGBGTY3LALK4DXWSKUTM2TFEYB4N5A3KNKW346V3F5XRN2Q, seg_pos={0 0}, upload=533792]: Done uploading piece. err: <nil> Dec 31 00:33:27.118759862 (609h4m26.989842897s) [noise=true , address=77.90.31.13:4632 , piece_id=5KUEGWEJV5NJE2EX3S5S2G7TS5TK23ZQXV3WPA4LLHO65QJRVW6Q, seg_pos={0 0}, upload=533792]: Done uploading piece. err: <nil> Dec 31 00:33:27.225324635 (609h4m27.09640767s) [noise=true , address=45.91.116.13:4161 , piece_id=K4CTNQGHL5S2GOARW2F42REGZ4QVWA2RKUNULONYX3VPVHAGONUQ, seg_pos={0 0}, upload=533792]: Done uploading piece. err: <nil> Dec 31 00:33:27.347765488 (609h4m27.218848533s) [noise=true , address=45.91.116.12:4248 , piece_id=QXKIUKBDX2GSJ2EQPALFCLPPLNWBPL3GRAEHLPJNSWUDIMU7WHYA, seg_pos={0 0}, upload=533792]: Done uploading piece. err: <nil> Dec 31 00:33:27.789508217 (609h4m27.660591252s) [noise=true , address=145.79.227.44:4217 , piece_id=GW44B36EHWVCRLEM6BA5MXC3ITRNCCJATW2RXSYTNC7K7WRO2IDQ, seg_pos={0 0}, upload=533792]: Done uploading piece. err: <nil> Dec 31 00:33:27.821042021 (609h4m27.692125056s) [noise=true , address=145.79.227.49:4353 , piece_id=MVTAW6QPJY2LRNEGISHNDP7AC5ZIIUR5N5BBZRFO5I6VYJU7UR3Q, seg_pos={0 0}, upload=533792]: Done uploading piece. err: <nil> Dec 31 00:33:27.828648224 (609h4m27.699731259s) [noise=true , address=145.79.227.49:4370 , piece_id=52YCNJY7BLJ3N3HDWBZCM7PP2ZPHHBS4GV4QS4WM2AWSLYKB5BBA, seg_pos={0 0}, upload=533792]: Done uploading piece. err: <nil> Dec 31 00:33:27.840271497 (609h4m27.711354542s) [noise=true , address=145.79.227.30:4769 , piece_id=P2ANRKDKCOTOMCZZMDIH2YRVI366UOHWO3LXV7INEDB2GW35Y52A, seg_pos={0 0}, upload=533792]: Done uploading piece. err: <nil> Dec 31 00:33:27.844064569 (609h4m27.715147614s) [noise=true , address=145.79.227.34:4930 , piece_id=L3AVRBWPDFTZANZKO4TGITCA3WHBECKEQ2BGMLFQJYN4CR7FSADA, seg_pos={0 0}, upload=533792]: Done uploading piece. err: <nil> Dec 31 00:33:27.867248213 (609h4m27.738331248s) [noise=true , address=77.90.31.18:4835 , piece_id=GRMJTN6ZDDYITHJWX5UGIAGLCOW6AKADHTK6UCVGRB437TUTDFGA, seg_pos={0 0}, upload=533792]: Done uploading piece. err: <nil> Dec 31 00:33:27.867270586 (609h4m27.738353621s) [seg_pos={0 0}, upload=533792]: Segment reached optimal threshold of 14 pieces. Dec 31 00:33:27.867476859 (609h4m27.738559904s) [noise=true , address=64.49.13.155:4913 , piece_id=E4WJCMYJVURPHENWEWR46KOAUAOZU7FW7WLC53DAU44BE7R3CHMQ, seg_pos={0 0}, upload=533792]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oyYUnWDyFo1npaKpBxMG5K31oUzJnTpcU1XHFmKux3MCVUPZV): protocol: expected piece hash; context canceled; EOF Dec 31 00:33:27.867485555 (609h4m27.73856859s) [noise=true , address=64.49.13.155:4913 , piece_id=E4WJCMYJVURPHENWEWR46KOAUAOZU7FW7WLC53DAU44BE7R3CHMQ, seg_pos={0 0}, upload=533792]: Long tail cancellation due to optimal threshold Dec 31 00:33:27.867516204 (609h4m27.738599249s) [noise=true , address=145.79.227.25:4630 , piece_id=5BVJWWAW2QUQABWWZCD4EW54ZVH3K5RJNM5EXJJUGXI33O6CC24A, seg_pos={0 0}, upload=533792]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12aETKgTzKw8fyYqbeDA2JzfxZnWn12xUPYK1eRt6TvBPPdRJDp): protocol: expected piece hash; context canceled; EOF Dec 31 00:33:27.867526553 (609h4m27.738609588s) [noise=true , address=145.79.227.25:4630 , piece_id=5BVJWWAW2QUQABWWZCD4EW54ZVH3K5RJNM5EXJJUGXI33O6CC24A, seg_pos={0 0}, upload=533792]: Long tail cancellation due to optimal threshold Dec 31 00:33:27.867714432 (609h4m27.738797477s) [noise=true , address=145.79.227.4:4189 , piece_id=ZZF4DSIPQBWKBD45UR2J5P75U5VVMTVB54NEX5QKZV4K2WJZ43BQ, seg_pos={0 0}, upload=533792]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jH3757i2F3q18XSpoXznsysZPaXmEnqynpLPDupgRfxJrTgKv): protocol: write tcp 188.166.225.170:57528->145.79.227.4:4189: use of closed network connection; write tcp 188.166.225.170:57528->145.79.227.4:4189: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57528->145.79.227.4:4189: use of closed network connection Dec 31 00:33:27.867723689 (609h4m27.738806724s) [noise=true , address=145.79.227.4:4189 , piece_id=ZZF4DSIPQBWKBD45UR2J5P75U5VVMTVB54NEX5QKZV4K2WJZ43BQ, seg_pos={0 0}, upload=533792]: Long tail cancellation due to optimal threshold Dec 31 00:33:27.867744699 (609h4m27.738827734s) [noise=true , address=145.79.227.2:4128 , piece_id=DW37Q32FFAZ37KHMOII6XGCJVDO2XTKED5TLOKG65JULWTXJIQEA, seg_pos={0 0}, upload=533792]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1wABkAtAjoYkKyNoAiaVCxSvY6WgvHvAYFPCjpNojtpiYiGtDS): protocol: expected piece hash; context canceled; EOF Dec 31 00:33:27.867753616 (609h4m27.738836651s) [noise=true , address=145.79.227.2:4128 , piece_id=DW37Q32FFAZ37KHMOII6XGCJVDO2XTKED5TLOKG65JULWTXJIQEA, seg_pos={0 0}, upload=533792]: Long tail cancellation due to optimal threshold Dec 31 00:33:27.867777462 (609h4m27.738860507s) [seg_pos={0 0}, upload=533792]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 31 00:33:27.867853687 (609h4m27.738936722s) [upload=533792]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 00:33:27.867859989 (609h4m27.738943024s) [upload=533792]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 00:33:28.167335511 (609h4m28.038418546s) [upload=533792]: Done uploading segments... Dec 31 00:33:28.167355899 (609h4m28.038438934s) [upload=533792]: Upload finished. err: <nil> Duration: 30.790679307s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 4.79897591s trace: https://jaeger.dev.storj.io/trace/63d344eb9b98ed45 FAIL (2025-12-31 00:44:16 UTC) orbiter warning: operation: DeleteObject uplink: metaclient: context deadline exceeded Duration: 4.799005876s core uplink eu1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.......................................................X....] duration: 3.638282073s trace: https://jaeger.dev.storj.io/trace/2c1d476356ae4e72 FAIL (2025-12-31 00:44:02 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.638313141s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 40.815818386s trace: https://jaeger.dev.storj.io/trace/67f6164cdd15ab81 FAIL (2025-12-31 00:43:49 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== 31 00:33:25.026367244 (609h4m24.897450289s) [noise=true , address=145.79.227.2:4128 , piece_id=DW37Q32FFAZ37KHMOII6XGCJVDO2XTKED5TLOKG65JULWTXJIQEA, seg_pos={0 0}, upload=533792]: Uploading piece... Dec 31 00:33:25.026878247 (609h4m24.897961292s) [noise=true , address=145.79.227.44:4217 , piece_id=GW44B36EHWVCRLEM6BA5MXC3ITRNCCJATW2RXSYTNC7K7WRO2IDQ, seg_pos={0 0}, upload=533792]: Uploading piece... Dec 31 00:33:25.027303136 (609h4m24.898386171s) [noise=true , address=45.91.116.13:4161 , piece_id=K4CTNQGHL5S2GOARW2F42REGZ4QVWA2RKUNULONYX3VPVHAGONUQ, seg_pos={0 0}, upload=533792]: Uploading piece... Dec 31 00:33:25.027901605 (609h4m24.89898465s) [noise=true , address=77.90.31.13:4632 , piece_id=5KUEGWEJV5NJE2EX3S5S2G7TS5TK23ZQXV3WPA4LLHO65QJRVW6Q, seg_pos={0 0}, upload=533792]: Uploading piece... Dec 31 00:33:25.028257373 (609h4m24.899340408s) [noise=true , address=145.79.227.30:4769 , piece_id=P2ANRKDKCOTOMCZZMDIH2YRVI366UOHWO3LXV7INEDB2GW35Y52A, seg_pos={0 0}, upload=533792]: Uploading piece... Dec 31 00:33:25.028623710 (609h4m24.899706745s) [noise=true , address=145.79.227.34:4930 , piece_id=L3AVRBWPDFTZANZKO4TGITCA3WHBECKEQ2BGMLFQJYN4CR7FSADA, seg_pos={0 0}, upload=533792]: Uploading piece... Dec 31 00:33:25.028952256 (609h4m24.900035291s) [noise=true , address=145.79.227.25:4630 , piece_id=5BVJWWAW2QUQABWWZCD4EW54ZVH3K5RJNM5EXJJUGXI33O6CC24A, seg_pos={0 0}, upload=533792]: Uploading piece... Dec 31 00:33:25.029314766 (609h4m24.900397801s) [noise=true , address=145.79.227.29:4971 , piece_id=EIY6YRG5ZRMEW7IWE5X25RCTGNCU3HQ27O7Q434OAZ4EI335HU4A, seg_pos={0 0}, upload=533792]: Uploading piece... Dec 31 00:33:25.029659533 (609h4m24.900742578s) [noise=true , address=145.79.227.6:4283 , piece_id=MXSG7TVTQBR2AFONVXYWQB4DBAQEEXW226FDC6SSVG5TTWGPAWFQ, seg_pos={0 0}, upload=533792]: Uploading piece... Dec 31 00:33:25.030576118 (609h4m24.901659153s) [noise=true , address=77.90.31.18:4835 , piece_id=GRMJTN6ZDDYITHJWX5UGIAGLCOW6AKADHTK6UCVGRB437TUTDFGA, seg_pos={0 0}, upload=533792]: Uploading piece... Dec 31 00:33:25.031413452 (609h4m24.902496487s) [noise=true , address=64.49.13.155:4913 , piece_id=E4WJCMYJVURPHENWEWR46KOAUAOZU7FW7WLC53DAU44BE7R3CHMQ, seg_pos={0 0}, upload=533792]: Uploading piece... Dec 31 00:33:25.031987204 (609h4m24.903070239s) [noise=true , address=145.79.227.49:4370 , piece_id=52YCNJY7BLJ3N3HDWBZCM7PP2ZPHHBS4GV4QS4WM2AWSLYKB5BBA, seg_pos={0 0}, upload=533792]: Uploading piece... Dec 31 00:33:25.032849947 (609h4m24.903932993s) [noise=true , address=145.79.227.4:4189 , piece_id=ZZF4DSIPQBWKBD45UR2J5P75U5VVMTVB54NEX5QKZV4K2WJZ43BQ, seg_pos={0 0}, upload=533792]: Uploading piece... Dec 31 00:33:25.034222320 (609h4m24.905305355s) [noise=true , address=145.79.227.49:4353 , piece_id=MVTAW6QPJY2LRNEGISHNDP7AC5ZIIUR5N5BBZRFO5I6VYJU7UR3Q, seg_pos={0 0}, upload=533792]: Uploading piece... Dec 31 00:33:25.035359376 (609h4m24.906442411s) [noise=true , address=145.79.227.35:4494 , piece_id=EY2OJLQIF4CH7DQWJWXSBJMQI37NAXSBLQP4ZPMU3IBRVLWJVYWA, seg_pos={0 0}, upload=533792]: Uploading piece... Dec 31 00:33:25.035852344 (609h4m24.906935379s) [noise=true , address=145.79.227.33:4873 , piece_id=2DSEGSC5UWL76NEXSE7WXOGSQNRNXALAINY52TLQHB57LTS2R6ZQ, seg_pos={0 0}, upload=533792]: Uploading piece... Dec 31 00:33:26.491429069 (609h4m26.362512104s) [upload=533792]: Next returned nil segment Dec 31 00:33:26.491441904 (609h4m26.362524939s) [upload=533792]: Waiting for error group managing segments... Dec 31 00:33:26.876786140 (609h4m26.747869175s) [noise=true , address=145.79.227.6:4283 , piece_id=MXSG7TVTQBR2AFONVXYWQB4DBAQEEXW226FDC6SSVG5TTWGPAWFQ, seg_pos={0 0}, upload=533792]: Done uploading piece. err: <nil> Dec 31 00:33:26.895923271 (609h4m26.767006316s) [noise=true , address=145.79.227.35:4494 , piece_id=EY2OJLQIF4CH7DQWJWXSBJMQI37NAXSBLQP4ZPMU3IBRVLWJVYWA, seg_pos={0 0}, upload=533792]: Done uploading piece. err: <nil> Dec 31 00:33:26.903106888 (609h4m26.774189923s) [noise=true , address=145.79.227.33:4873 , piece_id=2DSEGSC5UWL76NEXSE7WXOGSQNRNXALAINY52TLQHB57LTS2R6ZQ, seg_pos={0 0}, upload=533792]: Done uploading piece. err: <nil> Dec 31 00:33:26.912484285 (609h4m26.78356733s) [noise=true , address=145.79.227.29:4971 , piece_id=EIY6YRG5ZRMEW7IWE5X25RCTGNCU3HQ27O7Q434OAZ4EI335HU4A, seg_pos={0 0}, upload=533792]: Done uploading piece. err: <nil> Dec 31 00:33:27.004280888 (609h4m26.875363933s) [noise=true , address=145.79.227.21:4495 , piece_id=YWETKWGBGTY3LALK4DXWSKUTM2TFEYB4N5A3KNKW346V3F5XRN2Q, seg_pos={0 0}, upload=533792]: Done uploading piece. err: <nil> Dec 31 00:33:27.118759862 (609h4m26.989842897s) [noise=true , address=77.90.31.13:4632 , piece_id=5KUEGWEJV5NJE2EX3S5S2G7TS5TK23ZQXV3WPA4LLHO65QJRVW6Q, seg_pos={0 0}, upload=533792]: Done uploading piece. err: <nil> Dec 31 00:33:27.225324635 (609h4m27.09640767s) [noise=true , address=45.91.116.13:4161 , piece_id=K4CTNQGHL5S2GOARW2F42REGZ4QVWA2RKUNULONYX3VPVHAGONUQ, seg_pos={0 0}, upload=533792]: Done uploading piece. err: <nil> Dec 31 00:33:27.347765488 (609h4m27.218848533s) [noise=true , address=45.91.116.12:4248 , piece_id=QXKIUKBDX2GSJ2EQPALFCLPPLNWBPL3GRAEHLPJNSWUDIMU7WHYA, seg_pos={0 0}, upload=533792]: Done uploading piece. err: <nil> Dec 31 00:33:27.789508217 (609h4m27.660591252s) [noise=true , address=145.79.227.44:4217 , piece_id=GW44B36EHWVCRLEM6BA5MXC3ITRNCCJATW2RXSYTNC7K7WRO2IDQ, seg_pos={0 0}, upload=533792]: Done uploading piece. err: <nil> Dec 31 00:33:27.821042021 (609h4m27.692125056s) [noise=true , address=145.79.227.49:4353 , piece_id=MVTAW6QPJY2LRNEGISHNDP7AC5ZIIUR5N5BBZRFO5I6VYJU7UR3Q, seg_pos={0 0}, upload=533792]: Done uploading piece. err: <nil> Dec 31 00:33:27.828648224 (609h4m27.699731259s) [noise=true , address=145.79.227.49:4370 , piece_id=52YCNJY7BLJ3N3HDWBZCM7PP2ZPHHBS4GV4QS4WM2AWSLYKB5BBA, seg_pos={0 0}, upload=533792]: Done uploading piece. err: <nil> Dec 31 00:33:27.840271497 (609h4m27.711354542s) [noise=true , address=145.79.227.30:4769 , piece_id=P2ANRKDKCOTOMCZZMDIH2YRVI366UOHWO3LXV7INEDB2GW35Y52A, seg_pos={0 0}, upload=533792]: Done uploading piece. err: <nil> Dec 31 00:33:27.844064569 (609h4m27.715147614s) [noise=true , address=145.79.227.34:4930 , piece_id=L3AVRBWPDFTZANZKO4TGITCA3WHBECKEQ2BGMLFQJYN4CR7FSADA, seg_pos={0 0}, upload=533792]: Done uploading piece. err: <nil> Dec 31 00:33:27.867248213 (609h4m27.738331248s) [noise=true , address=77.90.31.18:4835 , piece_id=GRMJTN6ZDDYITHJWX5UGIAGLCOW6AKADHTK6UCVGRB437TUTDFGA, seg_pos={0 0}, upload=533792]: Done uploading piece. err: <nil> Dec 31 00:33:27.867270586 (609h4m27.738353621s) [seg_pos={0 0}, upload=533792]: Segment reached optimal threshold of 14 pieces. Dec 31 00:33:27.867476859 (609h4m27.738559904s) [noise=true , address=64.49.13.155:4913 , piece_id=E4WJCMYJVURPHENWEWR46KOAUAOZU7FW7WLC53DAU44BE7R3CHMQ, seg_pos={0 0}, upload=533792]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oyYUnWDyFo1npaKpBxMG5K31oUzJnTpcU1XHFmKux3MCVUPZV): protocol: expected piece hash; context canceled; EOF Dec 31 00:33:27.867485555 (609h4m27.73856859s) [noise=true , address=64.49.13.155:4913 , piece_id=E4WJCMYJVURPHENWEWR46KOAUAOZU7FW7WLC53DAU44BE7R3CHMQ, seg_pos={0 0}, upload=533792]: Long tail cancellation due to optimal threshold Dec 31 00:33:27.867516204 (609h4m27.738599249s) [noise=true , address=145.79.227.25:4630 , piece_id=5BVJWWAW2QUQABWWZCD4EW54ZVH3K5RJNM5EXJJUGXI33O6CC24A, seg_pos={0 0}, upload=533792]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12aETKgTzKw8fyYqbeDA2JzfxZnWn12xUPYK1eRt6TvBPPdRJDp): protocol: expected piece hash; context canceled; EOF Dec 31 00:33:27.867526553 (609h4m27.738609588s) [noise=true , address=145.79.227.25:4630 , piece_id=5BVJWWAW2QUQABWWZCD4EW54ZVH3K5RJNM5EXJJUGXI33O6CC24A, seg_pos={0 0}, upload=533792]: Long tail cancellation due to optimal threshold Dec 31 00:33:27.867714432 (609h4m27.738797477s) [noise=true , address=145.79.227.4:4189 , piece_id=ZZF4DSIPQBWKBD45UR2J5P75U5VVMTVB54NEX5QKZV4K2WJZ43BQ, seg_pos={0 0}, upload=533792]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jH3757i2F3q18XSpoXznsysZPaXmEnqynpLPDupgRfxJrTgKv): protocol: write tcp 188.166.225.170:57528->145.79.227.4:4189: use of closed network connection; write tcp 188.166.225.170:57528->145.79.227.4:4189: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57528->145.79.227.4:4189: use of closed network connection Dec 31 00:33:27.867723689 (609h4m27.738806724s) [noise=true , address=145.79.227.4:4189 , piece_id=ZZF4DSIPQBWKBD45UR2J5P75U5VVMTVB54NEX5QKZV4K2WJZ43BQ, seg_pos={0 0}, upload=533792]: Long tail cancellation due to optimal threshold Dec 31 00:33:27.867744699 (609h4m27.738827734s) [noise=true , address=145.79.227.2:4128 , piece_id=DW37Q32FFAZ37KHMOII6XGCJVDO2XTKED5TLOKG65JULWTXJIQEA, seg_pos={0 0}, upload=533792]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1wABkAtAjoYkKyNoAiaVCxSvY6WgvHvAYFPCjpNojtpiYiGtDS): protocol: expected piece hash; context canceled; EOF Dec 31 00:33:27.867753616 (609h4m27.738836651s) [noise=true , address=145.79.227.2:4128 , piece_id=DW37Q32FFAZ37KHMOII6XGCJVDO2XTKED5TLOKG65JULWTXJIQEA, seg_pos={0 0}, upload=533792]: Long tail cancellation due to optimal threshold Dec 31 00:33:27.867777462 (609h4m27.738860507s) [seg_pos={0 0}, upload=533792]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 31 00:33:27.867853687 (609h4m27.738936722s) [upload=533792]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 00:33:27.867859989 (609h4m27.738943024s) [upload=533792]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 00:33:28.167335511 (609h4m28.038418546s) [upload=533792]: Done uploading segments... Dec 31 00:33:28.167355899 (609h4m28.038438934s) [upload=533792]: Upload finished. err: <nil> Duration: 40.815844036s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [.........X...X...........X........X..X....XX.XXX....X.......] duration: 5.928488706s trace: https://jaeger.dev.storj.io/trace/4804197dbd063942 FAIL (2025-12-31 00:43:48 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, Piece ID: QRFMMAPSLAYNXMN5JG2PKJ5DZUGPOXUSDBYRBGCYR34WT63APUOA) context deadline exceeded; piecestore: (Node ID: 12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f, Piece ID: WAZYSDAN5MSE3C3UGJ5BQ7GDOPLPAOB7YQTHYUTIE7OZWRIVEARA) context deadline exceeded ======LOGS====== Dec 31 00:37:21.551769648 (609h8m21.422852683s) [upload=533848]: Starting upload Dec 31 00:37:21.551853758 (609h8m21.422936803s) [upload=533848]: Done starting upload Dec 31 00:37:21.551965832 (609h8m21.423048887s) [upload=533848]: Uploading segments... Dec 31 00:37:21.552177856 (609h8m21.423260901s) [upload=533848]: Got next segment. Inline: false Dec 31 00:37:21.552220857 (609h8m21.423303902s) [upload=533848]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 00:37:21.552226658 (609h8m21.423309703s) [upload=533848]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 00:37:21.804536710 (609h8m21.675619755s) [seg_pos={0 0}, upload=533848]: Begin upload segment... Dec 31 00:37:21.804586926 (609h8m21.675669961s) [seg_pos={0 0}, upload=533848]: Done begin upload segment. Dec 31 00:37:21.804611863 (609h8m21.675694908s) [noise=true , address=77.90.31.18:4805 , piece_id=QRFMMAPSLAYNXMN5JG2PKJ5DZUGPOXUSDBYRBGCYR34WT63APUOA, seg_pos={0 0}, upload=533848]: Uploading piece... Dec 31 00:37:21.804621622 (609h8m21.675704657s) [noise=true , address=77.90.31.21:4878 , piece_id=T6MNCWRE4CB45ARESCQV3UCXA6FHSBVSC6SFJCRA4FOKL4QJ7Y2A, seg_pos={0 0}, upload=533848]: Uploading piece... Dec 31 00:37:21.805095093 (609h8m21.676178138s) [noise=true , address=145.79.227.11:4732 , piece_id=NXZGC73AGULETYQ2EMVQ7XM36TNFNECSTUGTI3VMCPXJGGY2ALQA, seg_pos={0 0}, upload=533848]: Uploading piece... Dec 31 00:37:21.805232335 (609h8m21.67631537s) [noise=true , address=45.91.116.5:4299 , piece_id=2GPR5IUAX2AHYXUSSLGXOOU6C3PYEPIY6P4B67GBTMIE5X4FFVHQ, seg_pos={0 0}, upload=533848]: Uploading piece... Dec 31 00:37:21.805605015 (609h8m21.67668806s) [noise=true , address=145.79.227.41:4086 , piece_id=KCXGGAOKZNWFLLBV7I6P7LHVNM6DFRN5DITCURNMWBA2VE7CKVHA, seg_pos={0 0}, upload=533848]: Uploading piece... Dec 31 00:37:21.805935163 (609h8m21.677018198s) [noise=true , address=45.91.116.6:4975 , piece_id=AC5RI6DJ3L5QPO22FF3J3KQW2UEQVC5QBD247E35JIMIIQ5OQP4A, seg_pos={0 0}, upload=533848]: Uploading piece... Dec 31 00:37:21.805997943 (609h8m21.677080988s) [noise=true , address=45.91.116.8:4045 , piece_id=RTVLMBX2SLOK23HMAWEA4G6WHEEZEBH6XQRKKRK6WD5GD6BDW7NA, seg_pos={0 0}, upload=533848]: Uploading piece... Dec 31 00:37:21.806193325 (609h8m21.67727636s) [noise=true , address=145.79.227.31:4797 , piece_id=IZMS3LM7IG3RBXIMQTRDCINIQBQ4MNQM4XXDSJUTEJBBVTMSEZEA, seg_pos={0 0}, upload=533848]: Uploading piece... Dec 31 00:37:21.806390720 (609h8m21.677473765s) [noise=true , address=45.91.116.10:4081 , piece_id=6GJLWPKDJL6TUXLQLGS62F7YG6PPZ42O3VQT6KQFNTTBOZVSFDGQ, seg_pos={0 0}, upload=533848]: Uploading piece... Dec 31 00:37:21.806476133 (609h8m21.677559178s) [noise=true , address=145.79.227.43:4157 , piece_id=42TVBZYTZ4O3AJ5FHBZMPBMWMEUIZDGY5NXYHAZ3TBKNHG6QCNRQ, seg_pos={0 0}, upload=533848]: Uploading piece... Dec 31 00:37:21.806866076 (609h8m21.677949121s) [noise=true , address=145.79.227.19:4079 , piece_id=W2AGJA3O2XARCZEI5USBHZE6EIWIDAFU24WKPCUHQII65YQ2VAKA, seg_pos={0 0}, upload=533848]: Uploading piece... Dec 31 00:37:21.806964183 (609h8m21.678047228s) [noise=true , address=77.90.31.13:4629 , piece_id=WAZYSDAN5MSE3C3UGJ5BQ7GDOPLPAOB7YQTHYUTIE7OZWRIVEARA, seg_pos={0 0}, upload=533848]: Uploading piece... Dec 31 00:37:21.807173682 (609h8m21.678256717s) [noise=true , address=145.79.227.10:4402 , piece_id=UJXD3JBNBCKBK2H4OAW4KBCFXMZMP3MHDKYZOYFYSKCAJDWIU4YQ, seg_pos={0 0}, upload=533848]: Uploading piece... Dec 31 00:37:21.807465367 (609h8m21.678548402s) [noise=true , address=45.91.116.9:4262 , piece_id=6FNAPYOYWGGJYCTPXCMWI7L325YOIP6RDE7UZGJUH5XZICAA4CTQ, seg_pos={0 0}, upload=533848]: Uploading piece... Dec 31 00:37:21.807912939 (609h8m21.678995984s) [noise=true , address=77.90.31.14:4663 , piece_id=BEL6CMHHCPH6PJBFQBY2P22GI2DSHLU6C2FI3KREEKFXVESBJROQ, seg_pos={0 0}, upload=533848]: Uploading piece... Dec 31 00:37:21.808755323 (609h8m21.679838368s) [noise=true , address=145.79.227.29:4942 , piece_id=3M3YMDPBFYAW2RZGTFMZLPVGDP7HY72DLHR2PALDI6FJFYDLHFAA, seg_pos={0 0}, upload=533848]: Uploading piece... Dec 31 00:37:23.506877615 (609h8m23.37796065s) [upload=533848]: Next returned nil segment Dec 31 00:37:23.506893356 (609h8m23.377976401s) [upload=533848]: Waiting for error group managing segments... Dec 31 00:37:23.787196661 (609h8m23.658279696s) [noise=true , address=145.79.227.19:4079 , piece_id=W2AGJA3O2XARCZEI5USBHZE6EIWIDAFU24WKPCUHQII65YQ2VAKA, seg_pos={0 0}, upload=533848]: Done uploading piece. err: <nil> Dec 31 00:37:23.794707001 (609h8m23.665790046s) [noise=true , address=145.79.227.31:4797 , piece_id=IZMS3LM7IG3RBXIMQTRDCINIQBQ4MNQM4XXDSJUTEJBBVTMSEZEA, seg_pos={0 0}, upload=533848]: Done uploading piece. err: <nil> Dec 31 00:37:23.802772670 (609h8m23.673855705s) [noise=true , address=145.79.227.10:4402 , piece_id=UJXD3JBNBCKBK2H4OAW4KBCFXMZMP3MHDKYZOYFYSKCAJDWIU4YQ, seg_pos={0 0}, upload=533848]: Done uploading piece. err: <nil> Dec 31 00:37:23.809582226 (609h8m23.680665261s) [noise=true , address=145.79.227.11:4732 , piece_id=NXZGC73AGULETYQ2EMVQ7XM36TNFNECSTUGTI3VMCPXJGGY2ALQA, seg_pos={0 0}, upload=533848]: Done uploading piece. err: <nil> Dec 31 00:37:23.879984451 (609h8m23.751067486s) [noise=true , address=145.79.227.41:4086 , piece_id=KCXGGAOKZNWFLLBV7I6P7LHVNM6DFRN5DITCURNMWBA2VE7CKVHA, seg_pos={0 0}, upload=533848]: Done uploading piece. err: <nil> Dec 31 00:37:23.982919423 (609h8m23.854002458s) [noise=true , address=77.90.31.13:4629 , piece_id=WAZYSDAN5MSE3C3UGJ5BQ7GDOPLPAOB7YQTHYUTIE7OZWRIVEARA, seg_pos={0 0}, upload=533848]: Done uploading piece. err: <nil> Dec 31 00:37:24.025039846 (609h8m23.896122881s) [noise=true , address=77.90.31.18:4805 , piece_id=QRFMMAPSLAYNXMN5JG2PKJ5DZUGPOXUSDBYRBGCYR34WT63APUOA, seg_pos={0 0}, upload=533848]: Done uploading piece. err: <nil> Dec 31 00:37:24.081686509 (609h8m23.952769695s) [noise=true , address=45.91.116.8:4045 , piece_id=RTVLMBX2SLOK23HMAWEA4G6WHEEZEBH6XQRKKRK6WD5GD6BDW7NA, seg_pos={0 0}, upload=533848]: Done uploading piece. err: <nil> Dec 31 00:37:24.116560253 (609h8m23.987643288s) [noise=true , address=45.91.116.5:4299 , piece_id=2GPR5IUAX2AHYXUSSLGXOOU6C3PYEPIY6P4B67GBTMIE5X4FFVHQ, seg_pos={0 0}, upload=533848]: Done uploading piece. err: <nil> Dec 31 00:37:24.152081901 (609h8m24.023164936s) [noise=true , address=45.91.116.10:4081 , piece_id=6GJLWPKDJL6TUXLQLGS62F7YG6PPZ42O3VQT6KQFNTTBOZVSFDGQ, seg_pos={0 0}, upload=533848]: Done uploading piece. err: <nil> Dec 31 00:37:24.167484239 (609h8m24.038567284s) [noise=true , address=77.90.31.14:4663 , piece_id=BEL6CMHHCPH6PJBFQBY2P22GI2DSHLU6C2FI3KREEKFXVESBJROQ, seg_pos={0 0}, upload=533848]: Done uploading piece. err: <nil> Dec 31 00:37:24.199729909 (609h8m24.070812944s) [noise=true , address=45.91.116.9:4262 , piece_id=6FNAPYOYWGGJYCTPXCMWI7L325YOIP6RDE7UZGJUH5XZICAA4CTQ, seg_pos={0 0}, upload=533848]: Done uploading piece. err: <nil> Dec 31 00:37:24.728312881 (609h8m24.599395926s) [noise=true , address=77.90.31.21:4878 , piece_id=T6MNCWRE4CB45ARESCQV3UCXA6FHSBVSC6SFJCRA4FOKL4QJ7Y2A, seg_pos={0 0}, upload=533848]: Done uploading piece. err: <nil> Dec 31 00:37:24.728331858 (609h8m24.599414893s) [seg_pos={0 0}, upload=533848]: Segment reached optimal threshold of 13 pieces. Dec 31 00:37:24.728520977 (609h8m24.599604013s) [noise=true , address=145.79.227.43:4157 , piece_id=42TVBZYTZ4O3AJ5FHBZMPBMWMEUIZDGY5NXYHAZ3TBKNHG6QCNRQ, seg_pos={0 0}, upload=533848]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZrzADk2mKG4R9tjxUKjQeJvb4WmfjLksGFyvstHRfYbBC7XNz): protocol: expected piece hash; context canceled; EOF Dec 31 00:37:24.728527470 (609h8m24.599610505s) [noise=true , address=145.79.227.43:4157 , piece_id=42TVBZYTZ4O3AJ5FHBZMPBMWMEUIZDGY5NXYHAZ3TBKNHG6QCNRQ, seg_pos={0 0}, upload=533848]: Long tail cancellation due to optimal threshold Dec 31 00:37:24.728572045 (609h8m24.59965508s) [noise=true , address=145.79.227.29:4942 , piece_id=3M3YMDPBFYAW2RZGTFMZLPVGDP7HY72DLHR2PALDI6FJFYDLHFAA, seg_pos={0 0}, upload=533848]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17o881pUMuG7K2dv83k7RkrJCrmM7kgUiWesgam4GEw21KVUn7): protocol: expected piece hash; context canceled; EOF Dec 31 00:37:24.728580772 (609h8m24.599663807s) [noise=true , address=145.79.227.29:4942 , piece_id=3M3YMDPBFYAW2RZGTFMZLPVGDP7HY72DLHR2PALDI6FJFYDLHFAA, seg_pos={0 0}, upload=533848]: Long tail cancellation due to optimal threshold Dec 31 00:37:24.728718213 (609h8m24.599801258s) [noise=true , address=45.91.116.6:4975 , piece_id=AC5RI6DJ3L5QPO22FF3J3KQW2UEQVC5QBD247E35JIMIIQ5OQP4A, seg_pos={0 0}, upload=533848]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Dec 31 00:37:24.728724195 (609h8m24.59980723s) [noise=true , address=45.91.116.6:4975 , piece_id=AC5RI6DJ3L5QPO22FF3J3KQW2UEQVC5QBD247E35JIMIIQ5OQP4A, seg_pos={0 0}, upload=533848]: Long tail cancellation due to optimal threshold Dec 31 00:37:24.728742089 (609h8m24.599825124s) [seg_pos={0 0}, upload=533848]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 00:37:24.728824065 (609h8m24.5999071s) [upload=533848]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 00:37:24.728828854 (609h8m24.599911889s) [upload=533848]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 00:37:25.041591605 (609h8m24.91267466s) [upload=533848]: Done uploading segments... Dec 31 00:37:25.041615270 (609h8m24.912698315s) [upload=533848]: Upload finished. err: <nil> Duration: 5.928518414s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [........................................................X...] duration: 3.447842643s trace: https://jaeger.dev.storj.io/trace/2c1d476356ae4e72 FAIL (2025-12-31 00:43:28 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.447870536s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [..........X...X...........X........X..X....XX.XXX....X......] duration: 29.09331806s trace: https://jaeger.dev.storj.io/trace/4804197dbd063942 FAIL (2025-12-31 00:43:23 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, Piece ID: QRFMMAPSLAYNXMN5JG2PKJ5DZUGPOXUSDBYRBGCYR34WT63APUOA) context deadline exceeded; piecestore: (Node ID: 12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f, Piece ID: WAZYSDAN5MSE3C3UGJ5BQ7GDOPLPAOB7YQTHYUTIE7OZWRIVEARA) context deadline exceeded ======LOGS====== Dec 31 00:37:21.551769648 (609h8m21.422852683s) [upload=533848]: Starting upload Dec 31 00:37:21.551853758 (609h8m21.422936803s) [upload=533848]: Done starting upload Dec 31 00:37:21.551965832 (609h8m21.423048887s) [upload=533848]: Uploading segments... Dec 31 00:37:21.552177856 (609h8m21.423260901s) [upload=533848]: Got next segment. Inline: false Dec 31 00:37:21.552220857 (609h8m21.423303902s) [upload=533848]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 00:37:21.552226658 (609h8m21.423309703s) [upload=533848]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 00:37:21.804536710 (609h8m21.675619755s) [seg_pos={0 0}, upload=533848]: Begin upload segment... Dec 31 00:37:21.804586926 (609h8m21.675669961s) [seg_pos={0 0}, upload=533848]: Done begin upload segment. Dec 31 00:37:21.804611863 (609h8m21.675694908s) [noise=true , address=77.90.31.18:4805 , piece_id=QRFMMAPSLAYNXMN5JG2PKJ5DZUGPOXUSDBYRBGCYR34WT63APUOA, seg_pos={0 0}, upload=533848]: Uploading piece... Dec 31 00:37:21.804621622 (609h8m21.675704657s) [noise=true , address=77.90.31.21:4878 , piece_id=T6MNCWRE4CB45ARESCQV3UCXA6FHSBVSC6SFJCRA4FOKL4QJ7Y2A, seg_pos={0 0}, upload=533848]: Uploading piece... Dec 31 00:37:21.805095093 (609h8m21.676178138s) [noise=true , address=145.79.227.11:4732 , piece_id=NXZGC73AGULETYQ2EMVQ7XM36TNFNECSTUGTI3VMCPXJGGY2ALQA, seg_pos={0 0}, upload=533848]: Uploading piece... Dec 31 00:37:21.805232335 (609h8m21.67631537s) [noise=true , address=45.91.116.5:4299 , piece_id=2GPR5IUAX2AHYXUSSLGXOOU6C3PYEPIY6P4B67GBTMIE5X4FFVHQ, seg_pos={0 0}, upload=533848]: Uploading piece... Dec 31 00:37:21.805605015 (609h8m21.67668806s) [noise=true , address=145.79.227.41:4086 , piece_id=KCXGGAOKZNWFLLBV7I6P7LHVNM6DFRN5DITCURNMWBA2VE7CKVHA, seg_pos={0 0}, upload=533848]: Uploading piece... Dec 31 00:37:21.805935163 (609h8m21.677018198s) [noise=true , address=45.91.116.6:4975 , piece_id=AC5RI6DJ3L5QPO22FF3J3KQW2UEQVC5QBD247E35JIMIIQ5OQP4A, seg_pos={0 0}, upload=533848]: Uploading piece... Dec 31 00:37:21.805997943 (609h8m21.677080988s) [noise=true , address=45.91.116.8:4045 , piece_id=RTVLMBX2SLOK23HMAWEA4G6WHEEZEBH6XQRKKRK6WD5GD6BDW7NA, seg_pos={0 0}, upload=533848]: Uploading piece... Dec 31 00:37:21.806193325 (609h8m21.67727636s) [noise=true , address=145.79.227.31:4797 , piece_id=IZMS3LM7IG3RBXIMQTRDCINIQBQ4MNQM4XXDSJUTEJBBVTMSEZEA, seg_pos={0 0}, upload=533848]: Uploading piece... Dec 31 00:37:21.806390720 (609h8m21.677473765s) [noise=true , address=45.91.116.10:4081 , piece_id=6GJLWPKDJL6TUXLQLGS62F7YG6PPZ42O3VQT6KQFNTTBOZVSFDGQ, seg_pos={0 0}, upload=533848]: Uploading piece... Dec 31 00:37:21.806476133 (609h8m21.677559178s) [noise=true , address=145.79.227.43:4157 , piece_id=42TVBZYTZ4O3AJ5FHBZMPBMWMEUIZDGY5NXYHAZ3TBKNHG6QCNRQ, seg_pos={0 0}, upload=533848]: Uploading piece... Dec 31 00:37:21.806866076 (609h8m21.677949121s) [noise=true , address=145.79.227.19:4079 , piece_id=W2AGJA3O2XARCZEI5USBHZE6EIWIDAFU24WKPCUHQII65YQ2VAKA, seg_pos={0 0}, upload=533848]: Uploading piece... Dec 31 00:37:21.806964183 (609h8m21.678047228s) [noise=true , address=77.90.31.13:4629 , piece_id=WAZYSDAN5MSE3C3UGJ5BQ7GDOPLPAOB7YQTHYUTIE7OZWRIVEARA, seg_pos={0 0}, upload=533848]: Uploading piece... Dec 31 00:37:21.807173682 (609h8m21.678256717s) [noise=true , address=145.79.227.10:4402 , piece_id=UJXD3JBNBCKBK2H4OAW4KBCFXMZMP3MHDKYZOYFYSKCAJDWIU4YQ, seg_pos={0 0}, upload=533848]: Uploading piece... Dec 31 00:37:21.807465367 (609h8m21.678548402s) [noise=true , address=45.91.116.9:4262 , piece_id=6FNAPYOYWGGJYCTPXCMWI7L325YOIP6RDE7UZGJUH5XZICAA4CTQ, seg_pos={0 0}, upload=533848]: Uploading piece... Dec 31 00:37:21.807912939 (609h8m21.678995984s) [noise=true , address=77.90.31.14:4663 , piece_id=BEL6CMHHCPH6PJBFQBY2P22GI2DSHLU6C2FI3KREEKFXVESBJROQ, seg_pos={0 0}, upload=533848]: Uploading piece... Dec 31 00:37:21.808755323 (609h8m21.679838368s) [noise=true , address=145.79.227.29:4942 , piece_id=3M3YMDPBFYAW2RZGTFMZLPVGDP7HY72DLHR2PALDI6FJFYDLHFAA, seg_pos={0 0}, upload=533848]: Uploading piece... Dec 31 00:37:23.506877615 (609h8m23.37796065s) [upload=533848]: Next returned nil segment Dec 31 00:37:23.506893356 (609h8m23.377976401s) [upload=533848]: Waiting for error group managing segments... Dec 31 00:37:23.787196661 (609h8m23.658279696s) [noise=true , address=145.79.227.19:4079 , piece_id=W2AGJA3O2XARCZEI5USBHZE6EIWIDAFU24WKPCUHQII65YQ2VAKA, seg_pos={0 0}, upload=533848]: Done uploading piece. err: <nil> Dec 31 00:37:23.794707001 (609h8m23.665790046s) [noise=true , address=145.79.227.31:4797 , piece_id=IZMS3LM7IG3RBXIMQTRDCINIQBQ4MNQM4XXDSJUTEJBBVTMSEZEA, seg_pos={0 0}, upload=533848]: Done uploading piece. err: <nil> Dec 31 00:37:23.802772670 (609h8m23.673855705s) [noise=true , address=145.79.227.10:4402 , piece_id=UJXD3JBNBCKBK2H4OAW4KBCFXMZMP3MHDKYZOYFYSKCAJDWIU4YQ, seg_pos={0 0}, upload=533848]: Done uploading piece. err: <nil> Dec 31 00:37:23.809582226 (609h8m23.680665261s) [noise=true , address=145.79.227.11:4732 , piece_id=NXZGC73AGULETYQ2EMVQ7XM36TNFNECSTUGTI3VMCPXJGGY2ALQA, seg_pos={0 0}, upload=533848]: Done uploading piece. err: <nil> Dec 31 00:37:23.879984451 (609h8m23.751067486s) [noise=true , address=145.79.227.41:4086 , piece_id=KCXGGAOKZNWFLLBV7I6P7LHVNM6DFRN5DITCURNMWBA2VE7CKVHA, seg_pos={0 0}, upload=533848]: Done uploading piece. err: <nil> Dec 31 00:37:23.982919423 (609h8m23.854002458s) [noise=true , address=77.90.31.13:4629 , piece_id=WAZYSDAN5MSE3C3UGJ5BQ7GDOPLPAOB7YQTHYUTIE7OZWRIVEARA, seg_pos={0 0}, upload=533848]: Done uploading piece. err: <nil> Dec 31 00:37:24.025039846 (609h8m23.896122881s) [noise=true , address=77.90.31.18:4805 , piece_id=QRFMMAPSLAYNXMN5JG2PKJ5DZUGPOXUSDBYRBGCYR34WT63APUOA, seg_pos={0 0}, upload=533848]: Done uploading piece. err: <nil> Dec 31 00:37:24.081686509 (609h8m23.952769695s) [noise=true , address=45.91.116.8:4045 , piece_id=RTVLMBX2SLOK23HMAWEA4G6WHEEZEBH6XQRKKRK6WD5GD6BDW7NA, seg_pos={0 0}, upload=533848]: Done uploading piece. err: <nil> Dec 31 00:37:24.116560253 (609h8m23.987643288s) [noise=true , address=45.91.116.5:4299 , piece_id=2GPR5IUAX2AHYXUSSLGXOOU6C3PYEPIY6P4B67GBTMIE5X4FFVHQ, seg_pos={0 0}, upload=533848]: Done uploading piece. err: <nil> Dec 31 00:37:24.152081901 (609h8m24.023164936s) [noise=true , address=45.91.116.10:4081 , piece_id=6GJLWPKDJL6TUXLQLGS62F7YG6PPZ42O3VQT6KQFNTTBOZVSFDGQ, seg_pos={0 0}, upload=533848]: Done uploading piece. err: <nil> Dec 31 00:37:24.167484239 (609h8m24.038567284s) [noise=true , address=77.90.31.14:4663 , piece_id=BEL6CMHHCPH6PJBFQBY2P22GI2DSHLU6C2FI3KREEKFXVESBJROQ, seg_pos={0 0}, upload=533848]: Done uploading piece. err: <nil> Dec 31 00:37:24.199729909 (609h8m24.070812944s) [noise=true , address=45.91.116.9:4262 , piece_id=6FNAPYOYWGGJYCTPXCMWI7L325YOIP6RDE7UZGJUH5XZICAA4CTQ, seg_pos={0 0}, upload=533848]: Done uploading piece. err: <nil> Dec 31 00:37:24.728312881 (609h8m24.599395926s) [noise=true , address=77.90.31.21:4878 , piece_id=T6MNCWRE4CB45ARESCQV3UCXA6FHSBVSC6SFJCRA4FOKL4QJ7Y2A, seg_pos={0 0}, upload=533848]: Done uploading piece. err: <nil> Dec 31 00:37:24.728331858 (609h8m24.599414893s) [seg_pos={0 0}, upload=533848]: Segment reached optimal threshold of 13 pieces. Dec 31 00:37:24.728520977 (609h8m24.599604013s) [noise=true , address=145.79.227.43:4157 , piece_id=42TVBZYTZ4O3AJ5FHBZMPBMWMEUIZDGY5NXYHAZ3TBKNHG6QCNRQ, seg_pos={0 0}, upload=533848]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZrzADk2mKG4R9tjxUKjQeJvb4WmfjLksGFyvstHRfYbBC7XNz): protocol: expected piece hash; context canceled; EOF Dec 31 00:37:24.728527470 (609h8m24.599610505s) [noise=true , address=145.79.227.43:4157 , piece_id=42TVBZYTZ4O3AJ5FHBZMPBMWMEUIZDGY5NXYHAZ3TBKNHG6QCNRQ, seg_pos={0 0}, upload=533848]: Long tail cancellation due to optimal threshold Dec 31 00:37:24.728572045 (609h8m24.59965508s) [noise=true , address=145.79.227.29:4942 , piece_id=3M3YMDPBFYAW2RZGTFMZLPVGDP7HY72DLHR2PALDI6FJFYDLHFAA, seg_pos={0 0}, upload=533848]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17o881pUMuG7K2dv83k7RkrJCrmM7kgUiWesgam4GEw21KVUn7): protocol: expected piece hash; context canceled; EOF Dec 31 00:37:24.728580772 (609h8m24.599663807s) [noise=true , address=145.79.227.29:4942 , piece_id=3M3YMDPBFYAW2RZGTFMZLPVGDP7HY72DLHR2PALDI6FJFYDLHFAA, seg_pos={0 0}, upload=533848]: Long tail cancellation due to optimal threshold Dec 31 00:37:24.728718213 (609h8m24.599801258s) [noise=true , address=45.91.116.6:4975 , piece_id=AC5RI6DJ3L5QPO22FF3J3KQW2UEQVC5QBD247E35JIMIIQ5OQP4A, seg_pos={0 0}, upload=533848]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Dec 31 00:37:24.728724195 (609h8m24.59980723s) [noise=true , address=45.91.116.6:4975 , piece_id=AC5RI6DJ3L5QPO22FF3J3KQW2UEQVC5QBD247E35JIMIIQ5OQP4A, seg_pos={0 0}, upload=533848]: Long tail cancellation due to optimal threshold Dec 31 00:37:24.728742089 (609h8m24.599825124s) [seg_pos={0 0}, upload=533848]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 00:37:24.728824065 (609h8m24.5999071s) [upload=533848]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 00:37:24.728828854 (609h8m24.599911889s) [upload=533848]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 00:37:25.041591605 (609h8m24.91267466s) [upload=533848]: Done uploading segments... Dec 31 00:37:25.041615270 (609h8m24.912698315s) [upload=533848]: Upload finished. err: <nil> Duration: 29.093346794s core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 5.015641317s trace: https://jaeger.dev.storj.io/trace/63d344eb9b98ed45 FAIL (2025-12-31 00:43:17 UTC) orbiter warning: operation: DeleteObject uplink: metaclient: context deadline exceeded Duration: 5.015666696s core uplink eu1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.........................................................X..] duration: 3.473872627s trace: https://jaeger.dev.storj.io/trace/2c1d476356ae4e72 FAIL (2025-12-31 00:42:59 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.473928203s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 5.003531116s trace: https://jaeger.dev.storj.io/trace/63d344eb9b98ed45 FAIL (2025-12-31 00:42:41 UTC) orbiter warning: operation: DeleteObject uplink: metaclient: context deadline exceeded Duration: 5.00355954s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 18.671832317s trace: https://jaeger.dev.storj.io/trace/67f6164cdd15ab81 FAIL (2025-12-31 00:42:32 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== 31 00:33:25.026367244 (609h4m24.897450289s) [noise=true , address=145.79.227.2:4128 , piece_id=DW37Q32FFAZ37KHMOII6XGCJVDO2XTKED5TLOKG65JULWTXJIQEA, seg_pos={0 0}, upload=533792]: Uploading piece... Dec 31 00:33:25.026878247 (609h4m24.897961292s) [noise=true , address=145.79.227.44:4217 , piece_id=GW44B36EHWVCRLEM6BA5MXC3ITRNCCJATW2RXSYTNC7K7WRO2IDQ, seg_pos={0 0}, upload=533792]: Uploading piece... Dec 31 00:33:25.027303136 (609h4m24.898386171s) [noise=true , address=45.91.116.13:4161 , piece_id=K4CTNQGHL5S2GOARW2F42REGZ4QVWA2RKUNULONYX3VPVHAGONUQ, seg_pos={0 0}, upload=533792]: Uploading piece... Dec 31 00:33:25.027901605 (609h4m24.89898465s) [noise=true , address=77.90.31.13:4632 , piece_id=5KUEGWEJV5NJE2EX3S5S2G7TS5TK23ZQXV3WPA4LLHO65QJRVW6Q, seg_pos={0 0}, upload=533792]: Uploading piece... Dec 31 00:33:25.028257373 (609h4m24.899340408s) [noise=true , address=145.79.227.30:4769 , piece_id=P2ANRKDKCOTOMCZZMDIH2YRVI366UOHWO3LXV7INEDB2GW35Y52A, seg_pos={0 0}, upload=533792]: Uploading piece... Dec 31 00:33:25.028623710 (609h4m24.899706745s) [noise=true , address=145.79.227.34:4930 , piece_id=L3AVRBWPDFTZANZKO4TGITCA3WHBECKEQ2BGMLFQJYN4CR7FSADA, seg_pos={0 0}, upload=533792]: Uploading piece... Dec 31 00:33:25.028952256 (609h4m24.900035291s) [noise=true , address=145.79.227.25:4630 , piece_id=5BVJWWAW2QUQABWWZCD4EW54ZVH3K5RJNM5EXJJUGXI33O6CC24A, seg_pos={0 0}, upload=533792]: Uploading piece... Dec 31 00:33:25.029314766 (609h4m24.900397801s) [noise=true , address=145.79.227.29:4971 , piece_id=EIY6YRG5ZRMEW7IWE5X25RCTGNCU3HQ27O7Q434OAZ4EI335HU4A, seg_pos={0 0}, upload=533792]: Uploading piece... Dec 31 00:33:25.029659533 (609h4m24.900742578s) [noise=true , address=145.79.227.6:4283 , piece_id=MXSG7TVTQBR2AFONVXYWQB4DBAQEEXW226FDC6SSVG5TTWGPAWFQ, seg_pos={0 0}, upload=533792]: Uploading piece... Dec 31 00:33:25.030576118 (609h4m24.901659153s) [noise=true , address=77.90.31.18:4835 , piece_id=GRMJTN6ZDDYITHJWX5UGIAGLCOW6AKADHTK6UCVGRB437TUTDFGA, seg_pos={0 0}, upload=533792]: Uploading piece... Dec 31 00:33:25.031413452 (609h4m24.902496487s) [noise=true , address=64.49.13.155:4913 , piece_id=E4WJCMYJVURPHENWEWR46KOAUAOZU7FW7WLC53DAU44BE7R3CHMQ, seg_pos={0 0}, upload=533792]: Uploading piece... Dec 31 00:33:25.031987204 (609h4m24.903070239s) [noise=true , address=145.79.227.49:4370 , piece_id=52YCNJY7BLJ3N3HDWBZCM7PP2ZPHHBS4GV4QS4WM2AWSLYKB5BBA, seg_pos={0 0}, upload=533792]: Uploading piece... Dec 31 00:33:25.032849947 (609h4m24.903932993s) [noise=true , address=145.79.227.4:4189 , piece_id=ZZF4DSIPQBWKBD45UR2J5P75U5VVMTVB54NEX5QKZV4K2WJZ43BQ, seg_pos={0 0}, upload=533792]: Uploading piece... Dec 31 00:33:25.034222320 (609h4m24.905305355s) [noise=true , address=145.79.227.49:4353 , piece_id=MVTAW6QPJY2LRNEGISHNDP7AC5ZIIUR5N5BBZRFO5I6VYJU7UR3Q, seg_pos={0 0}, upload=533792]: Uploading piece... Dec 31 00:33:25.035359376 (609h4m24.906442411s) [noise=true , address=145.79.227.35:4494 , piece_id=EY2OJLQIF4CH7DQWJWXSBJMQI37NAXSBLQP4ZPMU3IBRVLWJVYWA, seg_pos={0 0}, upload=533792]: Uploading piece... Dec 31 00:33:25.035852344 (609h4m24.906935379s) [noise=true , address=145.79.227.33:4873 , piece_id=2DSEGSC5UWL76NEXSE7WXOGSQNRNXALAINY52TLQHB57LTS2R6ZQ, seg_pos={0 0}, upload=533792]: Uploading piece... Dec 31 00:33:26.491429069 (609h4m26.362512104s) [upload=533792]: Next returned nil segment Dec 31 00:33:26.491441904 (609h4m26.362524939s) [upload=533792]: Waiting for error group managing segments... Dec 31 00:33:26.876786140 (609h4m26.747869175s) [noise=true , address=145.79.227.6:4283 , piece_id=MXSG7TVTQBR2AFONVXYWQB4DBAQEEXW226FDC6SSVG5TTWGPAWFQ, seg_pos={0 0}, upload=533792]: Done uploading piece. err: <nil> Dec 31 00:33:26.895923271 (609h4m26.767006316s) [noise=true , address=145.79.227.35:4494 , piece_id=EY2OJLQIF4CH7DQWJWXSBJMQI37NAXSBLQP4ZPMU3IBRVLWJVYWA, seg_pos={0 0}, upload=533792]: Done uploading piece. err: <nil> Dec 31 00:33:26.903106888 (609h4m26.774189923s) [noise=true , address=145.79.227.33:4873 , piece_id=2DSEGSC5UWL76NEXSE7WXOGSQNRNXALAINY52TLQHB57LTS2R6ZQ, seg_pos={0 0}, upload=533792]: Done uploading piece. err: <nil> Dec 31 00:33:26.912484285 (609h4m26.78356733s) [noise=true , address=145.79.227.29:4971 , piece_id=EIY6YRG5ZRMEW7IWE5X25RCTGNCU3HQ27O7Q434OAZ4EI335HU4A, seg_pos={0 0}, upload=533792]: Done uploading piece. err: <nil> Dec 31 00:33:27.004280888 (609h4m26.875363933s) [noise=true , address=145.79.227.21:4495 , piece_id=YWETKWGBGTY3LALK4DXWSKUTM2TFEYB4N5A3KNKW346V3F5XRN2Q, seg_pos={0 0}, upload=533792]: Done uploading piece. err: <nil> Dec 31 00:33:27.118759862 (609h4m26.989842897s) [noise=true , address=77.90.31.13:4632 , piece_id=5KUEGWEJV5NJE2EX3S5S2G7TS5TK23ZQXV3WPA4LLHO65QJRVW6Q, seg_pos={0 0}, upload=533792]: Done uploading piece. err: <nil> Dec 31 00:33:27.225324635 (609h4m27.09640767s) [noise=true , address=45.91.116.13:4161 , piece_id=K4CTNQGHL5S2GOARW2F42REGZ4QVWA2RKUNULONYX3VPVHAGONUQ, seg_pos={0 0}, upload=533792]: Done uploading piece. err: <nil> Dec 31 00:33:27.347765488 (609h4m27.218848533s) [noise=true , address=45.91.116.12:4248 , piece_id=QXKIUKBDX2GSJ2EQPALFCLPPLNWBPL3GRAEHLPJNSWUDIMU7WHYA, seg_pos={0 0}, upload=533792]: Done uploading piece. err: <nil> Dec 31 00:33:27.789508217 (609h4m27.660591252s) [noise=true , address=145.79.227.44:4217 , piece_id=GW44B36EHWVCRLEM6BA5MXC3ITRNCCJATW2RXSYTNC7K7WRO2IDQ, seg_pos={0 0}, upload=533792]: Done uploading piece. err: <nil> Dec 31 00:33:27.821042021 (609h4m27.692125056s) [noise=true , address=145.79.227.49:4353 , piece_id=MVTAW6QPJY2LRNEGISHNDP7AC5ZIIUR5N5BBZRFO5I6VYJU7UR3Q, seg_pos={0 0}, upload=533792]: Done uploading piece. err: <nil> Dec 31 00:33:27.828648224 (609h4m27.699731259s) [noise=true , address=145.79.227.49:4370 , piece_id=52YCNJY7BLJ3N3HDWBZCM7PP2ZPHHBS4GV4QS4WM2AWSLYKB5BBA, seg_pos={0 0}, upload=533792]: Done uploading piece. err: <nil> Dec 31 00:33:27.840271497 (609h4m27.711354542s) [noise=true , address=145.79.227.30:4769 , piece_id=P2ANRKDKCOTOMCZZMDIH2YRVI366UOHWO3LXV7INEDB2GW35Y52A, seg_pos={0 0}, upload=533792]: Done uploading piece. err: <nil> Dec 31 00:33:27.844064569 (609h4m27.715147614s) [noise=true , address=145.79.227.34:4930 , piece_id=L3AVRBWPDFTZANZKO4TGITCA3WHBECKEQ2BGMLFQJYN4CR7FSADA, seg_pos={0 0}, upload=533792]: Done uploading piece. err: <nil> Dec 31 00:33:27.867248213 (609h4m27.738331248s) [noise=true , address=77.90.31.18:4835 , piece_id=GRMJTN6ZDDYITHJWX5UGIAGLCOW6AKADHTK6UCVGRB437TUTDFGA, seg_pos={0 0}, upload=533792]: Done uploading piece. err: <nil> Dec 31 00:33:27.867270586 (609h4m27.738353621s) [seg_pos={0 0}, upload=533792]: Segment reached optimal threshold of 14 pieces. Dec 31 00:33:27.867476859 (609h4m27.738559904s) [noise=true , address=64.49.13.155:4913 , piece_id=E4WJCMYJVURPHENWEWR46KOAUAOZU7FW7WLC53DAU44BE7R3CHMQ, seg_pos={0 0}, upload=533792]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oyYUnWDyFo1npaKpBxMG5K31oUzJnTpcU1XHFmKux3MCVUPZV): protocol: expected piece hash; context canceled; EOF Dec 31 00:33:27.867485555 (609h4m27.73856859s) [noise=true , address=64.49.13.155:4913 , piece_id=E4WJCMYJVURPHENWEWR46KOAUAOZU7FW7WLC53DAU44BE7R3CHMQ, seg_pos={0 0}, upload=533792]: Long tail cancellation due to optimal threshold Dec 31 00:33:27.867516204 (609h4m27.738599249s) [noise=true , address=145.79.227.25:4630 , piece_id=5BVJWWAW2QUQABWWZCD4EW54ZVH3K5RJNM5EXJJUGXI33O6CC24A, seg_pos={0 0}, upload=533792]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12aETKgTzKw8fyYqbeDA2JzfxZnWn12xUPYK1eRt6TvBPPdRJDp): protocol: expected piece hash; context canceled; EOF Dec 31 00:33:27.867526553 (609h4m27.738609588s) [noise=true , address=145.79.227.25:4630 , piece_id=5BVJWWAW2QUQABWWZCD4EW54ZVH3K5RJNM5EXJJUGXI33O6CC24A, seg_pos={0 0}, upload=533792]: Long tail cancellation due to optimal threshold Dec 31 00:33:27.867714432 (609h4m27.738797477s) [noise=true , address=145.79.227.4:4189 , piece_id=ZZF4DSIPQBWKBD45UR2J5P75U5VVMTVB54NEX5QKZV4K2WJZ43BQ, seg_pos={0 0}, upload=533792]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jH3757i2F3q18XSpoXznsysZPaXmEnqynpLPDupgRfxJrTgKv): protocol: write tcp 188.166.225.170:57528->145.79.227.4:4189: use of closed network connection; write tcp 188.166.225.170:57528->145.79.227.4:4189: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57528->145.79.227.4:4189: use of closed network connection Dec 31 00:33:27.867723689 (609h4m27.738806724s) [noise=true , address=145.79.227.4:4189 , piece_id=ZZF4DSIPQBWKBD45UR2J5P75U5VVMTVB54NEX5QKZV4K2WJZ43BQ, seg_pos={0 0}, upload=533792]: Long tail cancellation due to optimal threshold Dec 31 00:33:27.867744699 (609h4m27.738827734s) [noise=true , address=145.79.227.2:4128 , piece_id=DW37Q32FFAZ37KHMOII6XGCJVDO2XTKED5TLOKG65JULWTXJIQEA, seg_pos={0 0}, upload=533792]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1wABkAtAjoYkKyNoAiaVCxSvY6WgvHvAYFPCjpNojtpiYiGtDS): protocol: expected piece hash; context canceled; EOF Dec 31 00:33:27.867753616 (609h4m27.738836651s) [noise=true , address=145.79.227.2:4128 , piece_id=DW37Q32FFAZ37KHMOII6XGCJVDO2XTKED5TLOKG65JULWTXJIQEA, seg_pos={0 0}, upload=533792]: Long tail cancellation due to optimal threshold Dec 31 00:33:27.867777462 (609h4m27.738860507s) [seg_pos={0 0}, upload=533792]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 31 00:33:27.867853687 (609h4m27.738936722s) [upload=533792]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 00:33:27.867859989 (609h4m27.738943024s) [upload=533792]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 00:33:28.167335511 (609h4m28.038418546s) [upload=533792]: Done uploading segments... Dec 31 00:33:28.167355899 (609h4m28.038438934s) [upload=533792]: Upload finished. err: <nil> Duration: 18.671850573s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 35.045042408s trace: https://jaeger.dev.storj.io/trace/67f6164cdd15ab81 FAIL (2025-12-31 00:42:13 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== 31 00:33:25.026367244 (609h4m24.897450289s) [noise=true , address=145.79.227.2:4128 , piece_id=DW37Q32FFAZ37KHMOII6XGCJVDO2XTKED5TLOKG65JULWTXJIQEA, seg_pos={0 0}, upload=533792]: Uploading piece... Dec 31 00:33:25.026878247 (609h4m24.897961292s) [noise=true , address=145.79.227.44:4217 , piece_id=GW44B36EHWVCRLEM6BA5MXC3ITRNCCJATW2RXSYTNC7K7WRO2IDQ, seg_pos={0 0}, upload=533792]: Uploading piece... Dec 31 00:33:25.027303136 (609h4m24.898386171s) [noise=true , address=45.91.116.13:4161 , piece_id=K4CTNQGHL5S2GOARW2F42REGZ4QVWA2RKUNULONYX3VPVHAGONUQ, seg_pos={0 0}, upload=533792]: Uploading piece... Dec 31 00:33:25.027901605 (609h4m24.89898465s) [noise=true , address=77.90.31.13:4632 , piece_id=5KUEGWEJV5NJE2EX3S5S2G7TS5TK23ZQXV3WPA4LLHO65QJRVW6Q, seg_pos={0 0}, upload=533792]: Uploading piece... Dec 31 00:33:25.028257373 (609h4m24.899340408s) [noise=true , address=145.79.227.30:4769 , piece_id=P2ANRKDKCOTOMCZZMDIH2YRVI366UOHWO3LXV7INEDB2GW35Y52A, seg_pos={0 0}, upload=533792]: Uploading piece... Dec 31 00:33:25.028623710 (609h4m24.899706745s) [noise=true , address=145.79.227.34:4930 , piece_id=L3AVRBWPDFTZANZKO4TGITCA3WHBECKEQ2BGMLFQJYN4CR7FSADA, seg_pos={0 0}, upload=533792]: Uploading piece... Dec 31 00:33:25.028952256 (609h4m24.900035291s) [noise=true , address=145.79.227.25:4630 , piece_id=5BVJWWAW2QUQABWWZCD4EW54ZVH3K5RJNM5EXJJUGXI33O6CC24A, seg_pos={0 0}, upload=533792]: Uploading piece... Dec 31 00:33:25.029314766 (609h4m24.900397801s) [noise=true , address=145.79.227.29:4971 , piece_id=EIY6YRG5ZRMEW7IWE5X25RCTGNCU3HQ27O7Q434OAZ4EI335HU4A, seg_pos={0 0}, upload=533792]: Uploading piece... Dec 31 00:33:25.029659533 (609h4m24.900742578s) [noise=true , address=145.79.227.6:4283 , piece_id=MXSG7TVTQBR2AFONVXYWQB4DBAQEEXW226FDC6SSVG5TTWGPAWFQ, seg_pos={0 0}, upload=533792]: Uploading piece... Dec 31 00:33:25.030576118 (609h4m24.901659153s) [noise=true , address=77.90.31.18:4835 , piece_id=GRMJTN6ZDDYITHJWX5UGIAGLCOW6AKADHTK6UCVGRB437TUTDFGA, seg_pos={0 0}, upload=533792]: Uploading piece... Dec 31 00:33:25.031413452 (609h4m24.902496487s) [noise=true , address=64.49.13.155:4913 , piece_id=E4WJCMYJVURPHENWEWR46KOAUAOZU7FW7WLC53DAU44BE7R3CHMQ, seg_pos={0 0}, upload=533792]: Uploading piece... Dec 31 00:33:25.031987204 (609h4m24.903070239s) [noise=true , address=145.79.227.49:4370 , piece_id=52YCNJY7BLJ3N3HDWBZCM7PP2ZPHHBS4GV4QS4WM2AWSLYKB5BBA, seg_pos={0 0}, upload=533792]: Uploading piece... Dec 31 00:33:25.032849947 (609h4m24.903932993s) [noise=true , address=145.79.227.4:4189 , piece_id=ZZF4DSIPQBWKBD45UR2J5P75U5VVMTVB54NEX5QKZV4K2WJZ43BQ, seg_pos={0 0}, upload=533792]: Uploading piece... Dec 31 00:33:25.034222320 (609h4m24.905305355s) [noise=true , address=145.79.227.49:4353 , piece_id=MVTAW6QPJY2LRNEGISHNDP7AC5ZIIUR5N5BBZRFO5I6VYJU7UR3Q, seg_pos={0 0}, upload=533792]: Uploading piece... Dec 31 00:33:25.035359376 (609h4m24.906442411s) [noise=true , address=145.79.227.35:4494 , piece_id=EY2OJLQIF4CH7DQWJWXSBJMQI37NAXSBLQP4ZPMU3IBRVLWJVYWA, seg_pos={0 0}, upload=533792]: Uploading piece... Dec 31 00:33:25.035852344 (609h4m24.906935379s) [noise=true , address=145.79.227.33:4873 , piece_id=2DSEGSC5UWL76NEXSE7WXOGSQNRNXALAINY52TLQHB57LTS2R6ZQ, seg_pos={0 0}, upload=533792]: Uploading piece... Dec 31 00:33:26.491429069 (609h4m26.362512104s) [upload=533792]: Next returned nil segment Dec 31 00:33:26.491441904 (609h4m26.362524939s) [upload=533792]: Waiting for error group managing segments... Dec 31 00:33:26.876786140 (609h4m26.747869175s) [noise=true , address=145.79.227.6:4283 , piece_id=MXSG7TVTQBR2AFONVXYWQB4DBAQEEXW226FDC6SSVG5TTWGPAWFQ, seg_pos={0 0}, upload=533792]: Done uploading piece. err: <nil> Dec 31 00:33:26.895923271 (609h4m26.767006316s) [noise=true , address=145.79.227.35:4494 , piece_id=EY2OJLQIF4CH7DQWJWXSBJMQI37NAXSBLQP4ZPMU3IBRVLWJVYWA, seg_pos={0 0}, upload=533792]: Done uploading piece. err: <nil> Dec 31 00:33:26.903106888 (609h4m26.774189923s) [noise=true , address=145.79.227.33:4873 , piece_id=2DSEGSC5UWL76NEXSE7WXOGSQNRNXALAINY52TLQHB57LTS2R6ZQ, seg_pos={0 0}, upload=533792]: Done uploading piece. err: <nil> Dec 31 00:33:26.912484285 (609h4m26.78356733s) [noise=true , address=145.79.227.29:4971 , piece_id=EIY6YRG5ZRMEW7IWE5X25RCTGNCU3HQ27O7Q434OAZ4EI335HU4A, seg_pos={0 0}, upload=533792]: Done uploading piece. err: <nil> Dec 31 00:33:27.004280888 (609h4m26.875363933s) [noise=true , address=145.79.227.21:4495 , piece_id=YWETKWGBGTY3LALK4DXWSKUTM2TFEYB4N5A3KNKW346V3F5XRN2Q, seg_pos={0 0}, upload=533792]: Done uploading piece. err: <nil> Dec 31 00:33:27.118759862 (609h4m26.989842897s) [noise=true , address=77.90.31.13:4632 , piece_id=5KUEGWEJV5NJE2EX3S5S2G7TS5TK23ZQXV3WPA4LLHO65QJRVW6Q, seg_pos={0 0}, upload=533792]: Done uploading piece. err: <nil> Dec 31 00:33:27.225324635 (609h4m27.09640767s) [noise=true , address=45.91.116.13:4161 , piece_id=K4CTNQGHL5S2GOARW2F42REGZ4QVWA2RKUNULONYX3VPVHAGONUQ, seg_pos={0 0}, upload=533792]: Done uploading piece. err: <nil> Dec 31 00:33:27.347765488 (609h4m27.218848533s) [noise=true , address=45.91.116.12:4248 , piece_id=QXKIUKBDX2GSJ2EQPALFCLPPLNWBPL3GRAEHLPJNSWUDIMU7WHYA, seg_pos={0 0}, upload=533792]: Done uploading piece. err: <nil> Dec 31 00:33:27.789508217 (609h4m27.660591252s) [noise=true , address=145.79.227.44:4217 , piece_id=GW44B36EHWVCRLEM6BA5MXC3ITRNCCJATW2RXSYTNC7K7WRO2IDQ, seg_pos={0 0}, upload=533792]: Done uploading piece. err: <nil> Dec 31 00:33:27.821042021 (609h4m27.692125056s) [noise=true , address=145.79.227.49:4353 , piece_id=MVTAW6QPJY2LRNEGISHNDP7AC5ZIIUR5N5BBZRFO5I6VYJU7UR3Q, seg_pos={0 0}, upload=533792]: Done uploading piece. err: <nil> Dec 31 00:33:27.828648224 (609h4m27.699731259s) [noise=true , address=145.79.227.49:4370 , piece_id=52YCNJY7BLJ3N3HDWBZCM7PP2ZPHHBS4GV4QS4WM2AWSLYKB5BBA, seg_pos={0 0}, upload=533792]: Done uploading piece. err: <nil> Dec 31 00:33:27.840271497 (609h4m27.711354542s) [noise=true , address=145.79.227.30:4769 , piece_id=P2ANRKDKCOTOMCZZMDIH2YRVI366UOHWO3LXV7INEDB2GW35Y52A, seg_pos={0 0}, upload=533792]: Done uploading piece. err: <nil> Dec 31 00:33:27.844064569 (609h4m27.715147614s) [noise=true , address=145.79.227.34:4930 , piece_id=L3AVRBWPDFTZANZKO4TGITCA3WHBECKEQ2BGMLFQJYN4CR7FSADA, seg_pos={0 0}, upload=533792]: Done uploading piece. err: <nil> Dec 31 00:33:27.867248213 (609h4m27.738331248s) [noise=true , address=77.90.31.18:4835 , piece_id=GRMJTN6ZDDYITHJWX5UGIAGLCOW6AKADHTK6UCVGRB437TUTDFGA, seg_pos={0 0}, upload=533792]: Done uploading piece. err: <nil> Dec 31 00:33:27.867270586 (609h4m27.738353621s) [seg_pos={0 0}, upload=533792]: Segment reached optimal threshold of 14 pieces. Dec 31 00:33:27.867476859 (609h4m27.738559904s) [noise=true , address=64.49.13.155:4913 , piece_id=E4WJCMYJVURPHENWEWR46KOAUAOZU7FW7WLC53DAU44BE7R3CHMQ, seg_pos={0 0}, upload=533792]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oyYUnWDyFo1npaKpBxMG5K31oUzJnTpcU1XHFmKux3MCVUPZV): protocol: expected piece hash; context canceled; EOF Dec 31 00:33:27.867485555 (609h4m27.73856859s) [noise=true , address=64.49.13.155:4913 , piece_id=E4WJCMYJVURPHENWEWR46KOAUAOZU7FW7WLC53DAU44BE7R3CHMQ, seg_pos={0 0}, upload=533792]: Long tail cancellation due to optimal threshold Dec 31 00:33:27.867516204 (609h4m27.738599249s) [noise=true , address=145.79.227.25:4630 , piece_id=5BVJWWAW2QUQABWWZCD4EW54ZVH3K5RJNM5EXJJUGXI33O6CC24A, seg_pos={0 0}, upload=533792]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12aETKgTzKw8fyYqbeDA2JzfxZnWn12xUPYK1eRt6TvBPPdRJDp): protocol: expected piece hash; context canceled; EOF Dec 31 00:33:27.867526553 (609h4m27.738609588s) [noise=true , address=145.79.227.25:4630 , piece_id=5BVJWWAW2QUQABWWZCD4EW54ZVH3K5RJNM5EXJJUGXI33O6CC24A, seg_pos={0 0}, upload=533792]: Long tail cancellation due to optimal threshold Dec 31 00:33:27.867714432 (609h4m27.738797477s) [noise=true , address=145.79.227.4:4189 , piece_id=ZZF4DSIPQBWKBD45UR2J5P75U5VVMTVB54NEX5QKZV4K2WJZ43BQ, seg_pos={0 0}, upload=533792]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jH3757i2F3q18XSpoXznsysZPaXmEnqynpLPDupgRfxJrTgKv): protocol: write tcp 188.166.225.170:57528->145.79.227.4:4189: use of closed network connection; write tcp 188.166.225.170:57528->145.79.227.4:4189: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57528->145.79.227.4:4189: use of closed network connection Dec 31 00:33:27.867723689 (609h4m27.738806724s) [noise=true , address=145.79.227.4:4189 , piece_id=ZZF4DSIPQBWKBD45UR2J5P75U5VVMTVB54NEX5QKZV4K2WJZ43BQ, seg_pos={0 0}, upload=533792]: Long tail cancellation due to optimal threshold Dec 31 00:33:27.867744699 (609h4m27.738827734s) [noise=true , address=145.79.227.2:4128 , piece_id=DW37Q32FFAZ37KHMOII6XGCJVDO2XTKED5TLOKG65JULWTXJIQEA, seg_pos={0 0}, upload=533792]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1wABkAtAjoYkKyNoAiaVCxSvY6WgvHvAYFPCjpNojtpiYiGtDS): protocol: expected piece hash; context canceled; EOF Dec 31 00:33:27.867753616 (609h4m27.738836651s) [noise=true , address=145.79.227.2:4128 , piece_id=DW37Q32FFAZ37KHMOII6XGCJVDO2XTKED5TLOKG65JULWTXJIQEA, seg_pos={0 0}, upload=533792]: Long tail cancellation due to optimal threshold Dec 31 00:33:27.867777462 (609h4m27.738860507s) [seg_pos={0 0}, upload=533792]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 31 00:33:27.867853687 (609h4m27.738936722s) [upload=533792]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 00:33:27.867859989 (609h4m27.738943024s) [upload=533792]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 00:33:28.167335511 (609h4m28.038418546s) [upload=533792]: Done uploading segments... Dec 31 00:33:28.167355899 (609h4m28.038438934s) [upload=533792]: Upload finished. err: <nil> Duration: 35.045057797s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [...........X...X...........X........X..X....XX.XXX....X.....] duration: 6.05660444s trace: https://jaeger.dev.storj.io/trace/4804197dbd063942 FAIL (2025-12-31 00:42:12 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, Piece ID: QRFMMAPSLAYNXMN5JG2PKJ5DZUGPOXUSDBYRBGCYR34WT63APUOA) context deadline exceeded; piecestore: (Node ID: 12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f, Piece ID: WAZYSDAN5MSE3C3UGJ5BQ7GDOPLPAOB7YQTHYUTIE7OZWRIVEARA) context deadline exceeded ======LOGS====== Dec 31 00:37:21.551769648 (609h8m21.422852683s) [upload=533848]: Starting upload Dec 31 00:37:21.551853758 (609h8m21.422936803s) [upload=533848]: Done starting upload Dec 31 00:37:21.551965832 (609h8m21.423048887s) [upload=533848]: Uploading segments... Dec 31 00:37:21.552177856 (609h8m21.423260901s) [upload=533848]: Got next segment. Inline: false Dec 31 00:37:21.552220857 (609h8m21.423303902s) [upload=533848]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 00:37:21.552226658 (609h8m21.423309703s) [upload=533848]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 00:37:21.804536710 (609h8m21.675619755s) [seg_pos={0 0}, upload=533848]: Begin upload segment... Dec 31 00:37:21.804586926 (609h8m21.675669961s) [seg_pos={0 0}, upload=533848]: Done begin upload segment. Dec 31 00:37:21.804611863 (609h8m21.675694908s) [noise=true , address=77.90.31.18:4805 , piece_id=QRFMMAPSLAYNXMN5JG2PKJ5DZUGPOXUSDBYRBGCYR34WT63APUOA, seg_pos={0 0}, upload=533848]: Uploading piece... Dec 31 00:37:21.804621622 (609h8m21.675704657s) [noise=true , address=77.90.31.21:4878 , piece_id=T6MNCWRE4CB45ARESCQV3UCXA6FHSBVSC6SFJCRA4FOKL4QJ7Y2A, seg_pos={0 0}, upload=533848]: Uploading piece... Dec 31 00:37:21.805095093 (609h8m21.676178138s) [noise=true , address=145.79.227.11:4732 , piece_id=NXZGC73AGULETYQ2EMVQ7XM36TNFNECSTUGTI3VMCPXJGGY2ALQA, seg_pos={0 0}, upload=533848]: Uploading piece... Dec 31 00:37:21.805232335 (609h8m21.67631537s) [noise=true , address=45.91.116.5:4299 , piece_id=2GPR5IUAX2AHYXUSSLGXOOU6C3PYEPIY6P4B67GBTMIE5X4FFVHQ, seg_pos={0 0}, upload=533848]: Uploading piece... Dec 31 00:37:21.805605015 (609h8m21.67668806s) [noise=true , address=145.79.227.41:4086 , piece_id=KCXGGAOKZNWFLLBV7I6P7LHVNM6DFRN5DITCURNMWBA2VE7CKVHA, seg_pos={0 0}, upload=533848]: Uploading piece... Dec 31 00:37:21.805935163 (609h8m21.677018198s) [noise=true , address=45.91.116.6:4975 , piece_id=AC5RI6DJ3L5QPO22FF3J3KQW2UEQVC5QBD247E35JIMIIQ5OQP4A, seg_pos={0 0}, upload=533848]: Uploading piece... Dec 31 00:37:21.805997943 (609h8m21.677080988s) [noise=true , address=45.91.116.8:4045 , piece_id=RTVLMBX2SLOK23HMAWEA4G6WHEEZEBH6XQRKKRK6WD5GD6BDW7NA, seg_pos={0 0}, upload=533848]: Uploading piece... Dec 31 00:37:21.806193325 (609h8m21.67727636s) [noise=true , address=145.79.227.31:4797 , piece_id=IZMS3LM7IG3RBXIMQTRDCINIQBQ4MNQM4XXDSJUTEJBBVTMSEZEA, seg_pos={0 0}, upload=533848]: Uploading piece... Dec 31 00:37:21.806390720 (609h8m21.677473765s) [noise=true , address=45.91.116.10:4081 , piece_id=6GJLWPKDJL6TUXLQLGS62F7YG6PPZ42O3VQT6KQFNTTBOZVSFDGQ, seg_pos={0 0}, upload=533848]: Uploading piece... Dec 31 00:37:21.806476133 (609h8m21.677559178s) [noise=true , address=145.79.227.43:4157 , piece_id=42TVBZYTZ4O3AJ5FHBZMPBMWMEUIZDGY5NXYHAZ3TBKNHG6QCNRQ, seg_pos={0 0}, upload=533848]: Uploading piece... Dec 31 00:37:21.806866076 (609h8m21.677949121s) [noise=true , address=145.79.227.19:4079 , piece_id=W2AGJA3O2XARCZEI5USBHZE6EIWIDAFU24WKPCUHQII65YQ2VAKA, seg_pos={0 0}, upload=533848]: Uploading piece... Dec 31 00:37:21.806964183 (609h8m21.678047228s) [noise=true , address=77.90.31.13:4629 , piece_id=WAZYSDAN5MSE3C3UGJ5BQ7GDOPLPAOB7YQTHYUTIE7OZWRIVEARA, seg_pos={0 0}, upload=533848]: Uploading piece... Dec 31 00:37:21.807173682 (609h8m21.678256717s) [noise=true , address=145.79.227.10:4402 , piece_id=UJXD3JBNBCKBK2H4OAW4KBCFXMZMP3MHDKYZOYFYSKCAJDWIU4YQ, seg_pos={0 0}, upload=533848]: Uploading piece... Dec 31 00:37:21.807465367 (609h8m21.678548402s) [noise=true , address=45.91.116.9:4262 , piece_id=6FNAPYOYWGGJYCTPXCMWI7L325YOIP6RDE7UZGJUH5XZICAA4CTQ, seg_pos={0 0}, upload=533848]: Uploading piece... Dec 31 00:37:21.807912939 (609h8m21.678995984s) [noise=true , address=77.90.31.14:4663 , piece_id=BEL6CMHHCPH6PJBFQBY2P22GI2DSHLU6C2FI3KREEKFXVESBJROQ, seg_pos={0 0}, upload=533848]: Uploading piece... Dec 31 00:37:21.808755323 (609h8m21.679838368s) [noise=true , address=145.79.227.29:4942 , piece_id=3M3YMDPBFYAW2RZGTFMZLPVGDP7HY72DLHR2PALDI6FJFYDLHFAA, seg_pos={0 0}, upload=533848]: Uploading piece... Dec 31 00:37:23.506877615 (609h8m23.37796065s) [upload=533848]: Next returned nil segment Dec 31 00:37:23.506893356 (609h8m23.377976401s) [upload=533848]: Waiting for error group managing segments... Dec 31 00:37:23.787196661 (609h8m23.658279696s) [noise=true , address=145.79.227.19:4079 , piece_id=W2AGJA3O2XARCZEI5USBHZE6EIWIDAFU24WKPCUHQII65YQ2VAKA, seg_pos={0 0}, upload=533848]: Done uploading piece. err: <nil> Dec 31 00:37:23.794707001 (609h8m23.665790046s) [noise=true , address=145.79.227.31:4797 , piece_id=IZMS3LM7IG3RBXIMQTRDCINIQBQ4MNQM4XXDSJUTEJBBVTMSEZEA, seg_pos={0 0}, upload=533848]: Done uploading piece. err: <nil> Dec 31 00:37:23.802772670 (609h8m23.673855705s) [noise=true , address=145.79.227.10:4402 , piece_id=UJXD3JBNBCKBK2H4OAW4KBCFXMZMP3MHDKYZOYFYSKCAJDWIU4YQ, seg_pos={0 0}, upload=533848]: Done uploading piece. err: <nil> Dec 31 00:37:23.809582226 (609h8m23.680665261s) [noise=true , address=145.79.227.11:4732 , piece_id=NXZGC73AGULETYQ2EMVQ7XM36TNFNECSTUGTI3VMCPXJGGY2ALQA, seg_pos={0 0}, upload=533848]: Done uploading piece. err: <nil> Dec 31 00:37:23.879984451 (609h8m23.751067486s) [noise=true , address=145.79.227.41:4086 , piece_id=KCXGGAOKZNWFLLBV7I6P7LHVNM6DFRN5DITCURNMWBA2VE7CKVHA, seg_pos={0 0}, upload=533848]: Done uploading piece. err: <nil> Dec 31 00:37:23.982919423 (609h8m23.854002458s) [noise=true , address=77.90.31.13:4629 , piece_id=WAZYSDAN5MSE3C3UGJ5BQ7GDOPLPAOB7YQTHYUTIE7OZWRIVEARA, seg_pos={0 0}, upload=533848]: Done uploading piece. err: <nil> Dec 31 00:37:24.025039846 (609h8m23.896122881s) [noise=true , address=77.90.31.18:4805 , piece_id=QRFMMAPSLAYNXMN5JG2PKJ5DZUGPOXUSDBYRBGCYR34WT63APUOA, seg_pos={0 0}, upload=533848]: Done uploading piece. err: <nil> Dec 31 00:37:24.081686509 (609h8m23.952769695s) [noise=true , address=45.91.116.8:4045 , piece_id=RTVLMBX2SLOK23HMAWEA4G6WHEEZEBH6XQRKKRK6WD5GD6BDW7NA, seg_pos={0 0}, upload=533848]: Done uploading piece. err: <nil> Dec 31 00:37:24.116560253 (609h8m23.987643288s) [noise=true , address=45.91.116.5:4299 , piece_id=2GPR5IUAX2AHYXUSSLGXOOU6C3PYEPIY6P4B67GBTMIE5X4FFVHQ, seg_pos={0 0}, upload=533848]: Done uploading piece. err: <nil> Dec 31 00:37:24.152081901 (609h8m24.023164936s) [noise=true , address=45.91.116.10:4081 , piece_id=6GJLWPKDJL6TUXLQLGS62F7YG6PPZ42O3VQT6KQFNTTBOZVSFDGQ, seg_pos={0 0}, upload=533848]: Done uploading piece. err: <nil> Dec 31 00:37:24.167484239 (609h8m24.038567284s) [noise=true , address=77.90.31.14:4663 , piece_id=BEL6CMHHCPH6PJBFQBY2P22GI2DSHLU6C2FI3KREEKFXVESBJROQ, seg_pos={0 0}, upload=533848]: Done uploading piece. err: <nil> Dec 31 00:37:24.199729909 (609h8m24.070812944s) [noise=true , address=45.91.116.9:4262 , piece_id=6FNAPYOYWGGJYCTPXCMWI7L325YOIP6RDE7UZGJUH5XZICAA4CTQ, seg_pos={0 0}, upload=533848]: Done uploading piece. err: <nil> Dec 31 00:37:24.728312881 (609h8m24.599395926s) [noise=true , address=77.90.31.21:4878 , piece_id=T6MNCWRE4CB45ARESCQV3UCXA6FHSBVSC6SFJCRA4FOKL4QJ7Y2A, seg_pos={0 0}, upload=533848]: Done uploading piece. err: <nil> Dec 31 00:37:24.728331858 (609h8m24.599414893s) [seg_pos={0 0}, upload=533848]: Segment reached optimal threshold of 13 pieces. Dec 31 00:37:24.728520977 (609h8m24.599604013s) [noise=true , address=145.79.227.43:4157 , piece_id=42TVBZYTZ4O3AJ5FHBZMPBMWMEUIZDGY5NXYHAZ3TBKNHG6QCNRQ, seg_pos={0 0}, upload=533848]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZrzADk2mKG4R9tjxUKjQeJvb4WmfjLksGFyvstHRfYbBC7XNz): protocol: expected piece hash; context canceled; EOF Dec 31 00:37:24.728527470 (609h8m24.599610505s) [noise=true , address=145.79.227.43:4157 , piece_id=42TVBZYTZ4O3AJ5FHBZMPBMWMEUIZDGY5NXYHAZ3TBKNHG6QCNRQ, seg_pos={0 0}, upload=533848]: Long tail cancellation due to optimal threshold Dec 31 00:37:24.728572045 (609h8m24.59965508s) [noise=true , address=145.79.227.29:4942 , piece_id=3M3YMDPBFYAW2RZGTFMZLPVGDP7HY72DLHR2PALDI6FJFYDLHFAA, seg_pos={0 0}, upload=533848]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17o881pUMuG7K2dv83k7RkrJCrmM7kgUiWesgam4GEw21KVUn7): protocol: expected piece hash; context canceled; EOF Dec 31 00:37:24.728580772 (609h8m24.599663807s) [noise=true , address=145.79.227.29:4942 , piece_id=3M3YMDPBFYAW2RZGTFMZLPVGDP7HY72DLHR2PALDI6FJFYDLHFAA, seg_pos={0 0}, upload=533848]: Long tail cancellation due to optimal threshold Dec 31 00:37:24.728718213 (609h8m24.599801258s) [noise=true , address=45.91.116.6:4975 , piece_id=AC5RI6DJ3L5QPO22FF3J3KQW2UEQVC5QBD247E35JIMIIQ5OQP4A, seg_pos={0 0}, upload=533848]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Dec 31 00:37:24.728724195 (609h8m24.59980723s) [noise=true , address=45.91.116.6:4975 , piece_id=AC5RI6DJ3L5QPO22FF3J3KQW2UEQVC5QBD247E35JIMIIQ5OQP4A, seg_pos={0 0}, upload=533848]: Long tail cancellation due to optimal threshold Dec 31 00:37:24.728742089 (609h8m24.599825124s) [seg_pos={0 0}, upload=533848]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 00:37:24.728824065 (609h8m24.5999071s) [upload=533848]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 00:37:24.728828854 (609h8m24.599911889s) [upload=533848]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 00:37:25.041591605 (609h8m24.91267466s) [upload=533848]: Done uploading segments... Dec 31 00:37:25.041615270 (609h8m24.912698315s) [upload=533848]: Upload finished. err: <nil> Duration: 6.05663018s core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 4.557330167s trace: https://jaeger.dev.storj.io/trace/63d344eb9b98ed45 FAIL (2025-12-31 00:42:05 UTC) orbiter warning: operation: DeleteObject uplink: metaclient: context deadline exceeded Duration: 4.557361878s core uplink eu1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..........................................................X.] duration: 3.746777603s trace: https://jaeger.dev.storj.io/trace/2c1d476356ae4e72 FAIL (2025-12-31 00:41:47 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.746795417s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 50.222065784s trace: https://jaeger.dev.storj.io/trace/67f6164cdd15ab81 FAIL (2025-12-31 00:41:38 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== 31 00:33:25.026367244 (609h4m24.897450289s) [noise=true , address=145.79.227.2:4128 , piece_id=DW37Q32FFAZ37KHMOII6XGCJVDO2XTKED5TLOKG65JULWTXJIQEA, seg_pos={0 0}, upload=533792]: Uploading piece... Dec 31 00:33:25.026878247 (609h4m24.897961292s) [noise=true , address=145.79.227.44:4217 , piece_id=GW44B36EHWVCRLEM6BA5MXC3ITRNCCJATW2RXSYTNC7K7WRO2IDQ, seg_pos={0 0}, upload=533792]: Uploading piece... Dec 31 00:33:25.027303136 (609h4m24.898386171s) [noise=true , address=45.91.116.13:4161 , piece_id=K4CTNQGHL5S2GOARW2F42REGZ4QVWA2RKUNULONYX3VPVHAGONUQ, seg_pos={0 0}, upload=533792]: Uploading piece... Dec 31 00:33:25.027901605 (609h4m24.89898465s) [noise=true , address=77.90.31.13:4632 , piece_id=5KUEGWEJV5NJE2EX3S5S2G7TS5TK23ZQXV3WPA4LLHO65QJRVW6Q, seg_pos={0 0}, upload=533792]: Uploading piece... Dec 31 00:33:25.028257373 (609h4m24.899340408s) [noise=true , address=145.79.227.30:4769 , piece_id=P2ANRKDKCOTOMCZZMDIH2YRVI366UOHWO3LXV7INEDB2GW35Y52A, seg_pos={0 0}, upload=533792]: Uploading piece... Dec 31 00:33:25.028623710 (609h4m24.899706745s) [noise=true , address=145.79.227.34:4930 , piece_id=L3AVRBWPDFTZANZKO4TGITCA3WHBECKEQ2BGMLFQJYN4CR7FSADA, seg_pos={0 0}, upload=533792]: Uploading piece... Dec 31 00:33:25.028952256 (609h4m24.900035291s) [noise=true , address=145.79.227.25:4630 , piece_id=5BVJWWAW2QUQABWWZCD4EW54ZVH3K5RJNM5EXJJUGXI33O6CC24A, seg_pos={0 0}, upload=533792]: Uploading piece... Dec 31 00:33:25.029314766 (609h4m24.900397801s) [noise=true , address=145.79.227.29:4971 , piece_id=EIY6YRG5ZRMEW7IWE5X25RCTGNCU3HQ27O7Q434OAZ4EI335HU4A, seg_pos={0 0}, upload=533792]: Uploading piece... Dec 31 00:33:25.029659533 (609h4m24.900742578s) [noise=true , address=145.79.227.6:4283 , piece_id=MXSG7TVTQBR2AFONVXYWQB4DBAQEEXW226FDC6SSVG5TTWGPAWFQ, seg_pos={0 0}, upload=533792]: Uploading piece... Dec 31 00:33:25.030576118 (609h4m24.901659153s) [noise=true , address=77.90.31.18:4835 , piece_id=GRMJTN6ZDDYITHJWX5UGIAGLCOW6AKADHTK6UCVGRB437TUTDFGA, seg_pos={0 0}, upload=533792]: Uploading piece... Dec 31 00:33:25.031413452 (609h4m24.902496487s) [noise=true , address=64.49.13.155:4913 , piece_id=E4WJCMYJVURPHENWEWR46KOAUAOZU7FW7WLC53DAU44BE7R3CHMQ, seg_pos={0 0}, upload=533792]: Uploading piece... Dec 31 00:33:25.031987204 (609h4m24.903070239s) [noise=true , address=145.79.227.49:4370 , piece_id=52YCNJY7BLJ3N3HDWBZCM7PP2ZPHHBS4GV4QS4WM2AWSLYKB5BBA, seg_pos={0 0}, upload=533792]: Uploading piece... Dec 31 00:33:25.032849947 (609h4m24.903932993s) [noise=true , address=145.79.227.4:4189 , piece_id=ZZF4DSIPQBWKBD45UR2J5P75U5VVMTVB54NEX5QKZV4K2WJZ43BQ, seg_pos={0 0}, upload=533792]: Uploading piece... Dec 31 00:33:25.034222320 (609h4m24.905305355s) [noise=true , address=145.79.227.49:4353 , piece_id=MVTAW6QPJY2LRNEGISHNDP7AC5ZIIUR5N5BBZRFO5I6VYJU7UR3Q, seg_pos={0 0}, upload=533792]: Uploading piece... Dec 31 00:33:25.035359376 (609h4m24.906442411s) [noise=true , address=145.79.227.35:4494 , piece_id=EY2OJLQIF4CH7DQWJWXSBJMQI37NAXSBLQP4ZPMU3IBRVLWJVYWA, seg_pos={0 0}, upload=533792]: Uploading piece... Dec 31 00:33:25.035852344 (609h4m24.906935379s) [noise=true , address=145.79.227.33:4873 , piece_id=2DSEGSC5UWL76NEXSE7WXOGSQNRNXALAINY52TLQHB57LTS2R6ZQ, seg_pos={0 0}, upload=533792]: Uploading piece... Dec 31 00:33:26.491429069 (609h4m26.362512104s) [upload=533792]: Next returned nil segment Dec 31 00:33:26.491441904 (609h4m26.362524939s) [upload=533792]: Waiting for error group managing segments... Dec 31 00:33:26.876786140 (609h4m26.747869175s) [noise=true , address=145.79.227.6:4283 , piece_id=MXSG7TVTQBR2AFONVXYWQB4DBAQEEXW226FDC6SSVG5TTWGPAWFQ, seg_pos={0 0}, upload=533792]: Done uploading piece. err: <nil> Dec 31 00:33:26.895923271 (609h4m26.767006316s) [noise=true , address=145.79.227.35:4494 , piece_id=EY2OJLQIF4CH7DQWJWXSBJMQI37NAXSBLQP4ZPMU3IBRVLWJVYWA, seg_pos={0 0}, upload=533792]: Done uploading piece. err: <nil> Dec 31 00:33:26.903106888 (609h4m26.774189923s) [noise=true , address=145.79.227.33:4873 , piece_id=2DSEGSC5UWL76NEXSE7WXOGSQNRNXALAINY52TLQHB57LTS2R6ZQ, seg_pos={0 0}, upload=533792]: Done uploading piece. err: <nil> Dec 31 00:33:26.912484285 (609h4m26.78356733s) [noise=true , address=145.79.227.29:4971 , piece_id=EIY6YRG5ZRMEW7IWE5X25RCTGNCU3HQ27O7Q434OAZ4EI335HU4A, seg_pos={0 0}, upload=533792]: Done uploading piece. err: <nil> Dec 31 00:33:27.004280888 (609h4m26.875363933s) [noise=true , address=145.79.227.21:4495 , piece_id=YWETKWGBGTY3LALK4DXWSKUTM2TFEYB4N5A3KNKW346V3F5XRN2Q, seg_pos={0 0}, upload=533792]: Done uploading piece. err: <nil> Dec 31 00:33:27.118759862 (609h4m26.989842897s) [noise=true , address=77.90.31.13:4632 , piece_id=5KUEGWEJV5NJE2EX3S5S2G7TS5TK23ZQXV3WPA4LLHO65QJRVW6Q, seg_pos={0 0}, upload=533792]: Done uploading piece. err: <nil> Dec 31 00:33:27.225324635 (609h4m27.09640767s) [noise=true , address=45.91.116.13:4161 , piece_id=K4CTNQGHL5S2GOARW2F42REGZ4QVWA2RKUNULONYX3VPVHAGONUQ, seg_pos={0 0}, upload=533792]: Done uploading piece. err: <nil> Dec 31 00:33:27.347765488 (609h4m27.218848533s) [noise=true , address=45.91.116.12:4248 , piece_id=QXKIUKBDX2GSJ2EQPALFCLPPLNWBPL3GRAEHLPJNSWUDIMU7WHYA, seg_pos={0 0}, upload=533792]: Done uploading piece. err: <nil> Dec 31 00:33:27.789508217 (609h4m27.660591252s) [noise=true , address=145.79.227.44:4217 , piece_id=GW44B36EHWVCRLEM6BA5MXC3ITRNCCJATW2RXSYTNC7K7WRO2IDQ, seg_pos={0 0}, upload=533792]: Done uploading piece. err: <nil> Dec 31 00:33:27.821042021 (609h4m27.692125056s) [noise=true , address=145.79.227.49:4353 , piece_id=MVTAW6QPJY2LRNEGISHNDP7AC5ZIIUR5N5BBZRFO5I6VYJU7UR3Q, seg_pos={0 0}, upload=533792]: Done uploading piece. err: <nil> Dec 31 00:33:27.828648224 (609h4m27.699731259s) [noise=true , address=145.79.227.49:4370 , piece_id=52YCNJY7BLJ3N3HDWBZCM7PP2ZPHHBS4GV4QS4WM2AWSLYKB5BBA, seg_pos={0 0}, upload=533792]: Done uploading piece. err: <nil> Dec 31 00:33:27.840271497 (609h4m27.711354542s) [noise=true , address=145.79.227.30:4769 , piece_id=P2ANRKDKCOTOMCZZMDIH2YRVI366UOHWO3LXV7INEDB2GW35Y52A, seg_pos={0 0}, upload=533792]: Done uploading piece. err: <nil> Dec 31 00:33:27.844064569 (609h4m27.715147614s) [noise=true , address=145.79.227.34:4930 , piece_id=L3AVRBWPDFTZANZKO4TGITCA3WHBECKEQ2BGMLFQJYN4CR7FSADA, seg_pos={0 0}, upload=533792]: Done uploading piece. err: <nil> Dec 31 00:33:27.867248213 (609h4m27.738331248s) [noise=true , address=77.90.31.18:4835 , piece_id=GRMJTN6ZDDYITHJWX5UGIAGLCOW6AKADHTK6UCVGRB437TUTDFGA, seg_pos={0 0}, upload=533792]: Done uploading piece. err: <nil> Dec 31 00:33:27.867270586 (609h4m27.738353621s) [seg_pos={0 0}, upload=533792]: Segment reached optimal threshold of 14 pieces. Dec 31 00:33:27.867476859 (609h4m27.738559904s) [noise=true , address=64.49.13.155:4913 , piece_id=E4WJCMYJVURPHENWEWR46KOAUAOZU7FW7WLC53DAU44BE7R3CHMQ, seg_pos={0 0}, upload=533792]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oyYUnWDyFo1npaKpBxMG5K31oUzJnTpcU1XHFmKux3MCVUPZV): protocol: expected piece hash; context canceled; EOF Dec 31 00:33:27.867485555 (609h4m27.73856859s) [noise=true , address=64.49.13.155:4913 , piece_id=E4WJCMYJVURPHENWEWR46KOAUAOZU7FW7WLC53DAU44BE7R3CHMQ, seg_pos={0 0}, upload=533792]: Long tail cancellation due to optimal threshold Dec 31 00:33:27.867516204 (609h4m27.738599249s) [noise=true , address=145.79.227.25:4630 , piece_id=5BVJWWAW2QUQABWWZCD4EW54ZVH3K5RJNM5EXJJUGXI33O6CC24A, seg_pos={0 0}, upload=533792]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12aETKgTzKw8fyYqbeDA2JzfxZnWn12xUPYK1eRt6TvBPPdRJDp): protocol: expected piece hash; context canceled; EOF Dec 31 00:33:27.867526553 (609h4m27.738609588s) [noise=true , address=145.79.227.25:4630 , piece_id=5BVJWWAW2QUQABWWZCD4EW54ZVH3K5RJNM5EXJJUGXI33O6CC24A, seg_pos={0 0}, upload=533792]: Long tail cancellation due to optimal threshold Dec 31 00:33:27.867714432 (609h4m27.738797477s) [noise=true , address=145.79.227.4:4189 , piece_id=ZZF4DSIPQBWKBD45UR2J5P75U5VVMTVB54NEX5QKZV4K2WJZ43BQ, seg_pos={0 0}, upload=533792]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jH3757i2F3q18XSpoXznsysZPaXmEnqynpLPDupgRfxJrTgKv): protocol: write tcp 188.166.225.170:57528->145.79.227.4:4189: use of closed network connection; write tcp 188.166.225.170:57528->145.79.227.4:4189: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57528->145.79.227.4:4189: use of closed network connection Dec 31 00:33:27.867723689 (609h4m27.738806724s) [noise=true , address=145.79.227.4:4189 , piece_id=ZZF4DSIPQBWKBD45UR2J5P75U5VVMTVB54NEX5QKZV4K2WJZ43BQ, seg_pos={0 0}, upload=533792]: Long tail cancellation due to optimal threshold Dec 31 00:33:27.867744699 (609h4m27.738827734s) [noise=true , address=145.79.227.2:4128 , piece_id=DW37Q32FFAZ37KHMOII6XGCJVDO2XTKED5TLOKG65JULWTXJIQEA, seg_pos={0 0}, upload=533792]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1wABkAtAjoYkKyNoAiaVCxSvY6WgvHvAYFPCjpNojtpiYiGtDS): protocol: expected piece hash; context canceled; EOF Dec 31 00:33:27.867753616 (609h4m27.738836651s) [noise=true , address=145.79.227.2:4128 , piece_id=DW37Q32FFAZ37KHMOII6XGCJVDO2XTKED5TLOKG65JULWTXJIQEA, seg_pos={0 0}, upload=533792]: Long tail cancellation due to optimal threshold Dec 31 00:33:27.867777462 (609h4m27.738860507s) [seg_pos={0 0}, upload=533792]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 31 00:33:27.867853687 (609h4m27.738936722s) [upload=533792]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 00:33:27.867859989 (609h4m27.738943024s) [upload=533792]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 00:33:28.167335511 (609h4m28.038418546s) [upload=533792]: Done uploading segments... Dec 31 00:33:28.167355899 (609h4m28.038438934s) [upload=533792]: Upload finished. err: <nil> Duration: 50.222090712s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...........................................................X] duration: 5.001689918s trace: https://jaeger.dev.storj.io/trace/2c1d476356ae4e72 FAIL (2025-12-31 00:41:27 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001711729s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [............X...X...........X........X..X....XX.XXX....X....] duration: 5.877189372s trace: https://jaeger.dev.storj.io/trace/4804197dbd063942 FAIL (2025-12-31 00:41:14 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, Piece ID: QRFMMAPSLAYNXMN5JG2PKJ5DZUGPOXUSDBYRBGCYR34WT63APUOA) context deadline exceeded; piecestore: (Node ID: 12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f, Piece ID: WAZYSDAN5MSE3C3UGJ5BQ7GDOPLPAOB7YQTHYUTIE7OZWRIVEARA) context deadline exceeded ======LOGS====== Dec 31 00:37:21.551769648 (609h8m21.422852683s) [upload=533848]: Starting upload Dec 31 00:37:21.551853758 (609h8m21.422936803s) [upload=533848]: Done starting upload Dec 31 00:37:21.551965832 (609h8m21.423048887s) [upload=533848]: Uploading segments... Dec 31 00:37:21.552177856 (609h8m21.423260901s) [upload=533848]: Got next segment. Inline: false Dec 31 00:37:21.552220857 (609h8m21.423303902s) [upload=533848]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 00:37:21.552226658 (609h8m21.423309703s) [upload=533848]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 00:37:21.804536710 (609h8m21.675619755s) [seg_pos={0 0}, upload=533848]: Begin upload segment... Dec 31 00:37:21.804586926 (609h8m21.675669961s) [seg_pos={0 0}, upload=533848]: Done begin upload segment. Dec 31 00:37:21.804611863 (609h8m21.675694908s) [noise=true , address=77.90.31.18:4805 , piece_id=QRFMMAPSLAYNXMN5JG2PKJ5DZUGPOXUSDBYRBGCYR34WT63APUOA, seg_pos={0 0}, upload=533848]: Uploading piece... Dec 31 00:37:21.804621622 (609h8m21.675704657s) [noise=true , address=77.90.31.21:4878 , piece_id=T6MNCWRE4CB45ARESCQV3UCXA6FHSBVSC6SFJCRA4FOKL4QJ7Y2A, seg_pos={0 0}, upload=533848]: Uploading piece... Dec 31 00:37:21.805095093 (609h8m21.676178138s) [noise=true , address=145.79.227.11:4732 , piece_id=NXZGC73AGULETYQ2EMVQ7XM36TNFNECSTUGTI3VMCPXJGGY2ALQA, seg_pos={0 0}, upload=533848]: Uploading piece... Dec 31 00:37:21.805232335 (609h8m21.67631537s) [noise=true , address=45.91.116.5:4299 , piece_id=2GPR5IUAX2AHYXUSSLGXOOU6C3PYEPIY6P4B67GBTMIE5X4FFVHQ, seg_pos={0 0}, upload=533848]: Uploading piece... Dec 31 00:37:21.805605015 (609h8m21.67668806s) [noise=true , address=145.79.227.41:4086 , piece_id=KCXGGAOKZNWFLLBV7I6P7LHVNM6DFRN5DITCURNMWBA2VE7CKVHA, seg_pos={0 0}, upload=533848]: Uploading piece... Dec 31 00:37:21.805935163 (609h8m21.677018198s) [noise=true , address=45.91.116.6:4975 , piece_id=AC5RI6DJ3L5QPO22FF3J3KQW2UEQVC5QBD247E35JIMIIQ5OQP4A, seg_pos={0 0}, upload=533848]: Uploading piece... Dec 31 00:37:21.805997943 (609h8m21.677080988s) [noise=true , address=45.91.116.8:4045 , piece_id=RTVLMBX2SLOK23HMAWEA4G6WHEEZEBH6XQRKKRK6WD5GD6BDW7NA, seg_pos={0 0}, upload=533848]: Uploading piece... Dec 31 00:37:21.806193325 (609h8m21.67727636s) [noise=true , address=145.79.227.31:4797 , piece_id=IZMS3LM7IG3RBXIMQTRDCINIQBQ4MNQM4XXDSJUTEJBBVTMSEZEA, seg_pos={0 0}, upload=533848]: Uploading piece... Dec 31 00:37:21.806390720 (609h8m21.677473765s) [noise=true , address=45.91.116.10:4081 , piece_id=6GJLWPKDJL6TUXLQLGS62F7YG6PPZ42O3VQT6KQFNTTBOZVSFDGQ, seg_pos={0 0}, upload=533848]: Uploading piece... Dec 31 00:37:21.806476133 (609h8m21.677559178s) [noise=true , address=145.79.227.43:4157 , piece_id=42TVBZYTZ4O3AJ5FHBZMPBMWMEUIZDGY5NXYHAZ3TBKNHG6QCNRQ, seg_pos={0 0}, upload=533848]: Uploading piece... Dec 31 00:37:21.806866076 (609h8m21.677949121s) [noise=true , address=145.79.227.19:4079 , piece_id=W2AGJA3O2XARCZEI5USBHZE6EIWIDAFU24WKPCUHQII65YQ2VAKA, seg_pos={0 0}, upload=533848]: Uploading piece... Dec 31 00:37:21.806964183 (609h8m21.678047228s) [noise=true , address=77.90.31.13:4629 , piece_id=WAZYSDAN5MSE3C3UGJ5BQ7GDOPLPAOB7YQTHYUTIE7OZWRIVEARA, seg_pos={0 0}, upload=533848]: Uploading piece... Dec 31 00:37:21.807173682 (609h8m21.678256717s) [noise=true , address=145.79.227.10:4402 , piece_id=UJXD3JBNBCKBK2H4OAW4KBCFXMZMP3MHDKYZOYFYSKCAJDWIU4YQ, seg_pos={0 0}, upload=533848]: Uploading piece... Dec 31 00:37:21.807465367 (609h8m21.678548402s) [noise=true , address=45.91.116.9:4262 , piece_id=6FNAPYOYWGGJYCTPXCMWI7L325YOIP6RDE7UZGJUH5XZICAA4CTQ, seg_pos={0 0}, upload=533848]: Uploading piece... Dec 31 00:37:21.807912939 (609h8m21.678995984s) [noise=true , address=77.90.31.14:4663 , piece_id=BEL6CMHHCPH6PJBFQBY2P22GI2DSHLU6C2FI3KREEKFXVESBJROQ, seg_pos={0 0}, upload=533848]: Uploading piece... Dec 31 00:37:21.808755323 (609h8m21.679838368s) [noise=true , address=145.79.227.29:4942 , piece_id=3M3YMDPBFYAW2RZGTFMZLPVGDP7HY72DLHR2PALDI6FJFYDLHFAA, seg_pos={0 0}, upload=533848]: Uploading piece... Dec 31 00:37:23.506877615 (609h8m23.37796065s) [upload=533848]: Next returned nil segment Dec 31 00:37:23.506893356 (609h8m23.377976401s) [upload=533848]: Waiting for error group managing segments... Dec 31 00:37:23.787196661 (609h8m23.658279696s) [noise=true , address=145.79.227.19:4079 , piece_id=W2AGJA3O2XARCZEI5USBHZE6EIWIDAFU24WKPCUHQII65YQ2VAKA, seg_pos={0 0}, upload=533848]: Done uploading piece. err: <nil> Dec 31 00:37:23.794707001 (609h8m23.665790046s) [noise=true , address=145.79.227.31:4797 , piece_id=IZMS3LM7IG3RBXIMQTRDCINIQBQ4MNQM4XXDSJUTEJBBVTMSEZEA, seg_pos={0 0}, upload=533848]: Done uploading piece. err: <nil> Dec 31 00:37:23.802772670 (609h8m23.673855705s) [noise=true , address=145.79.227.10:4402 , piece_id=UJXD3JBNBCKBK2H4OAW4KBCFXMZMP3MHDKYZOYFYSKCAJDWIU4YQ, seg_pos={0 0}, upload=533848]: Done uploading piece. err: <nil> Dec 31 00:37:23.809582226 (609h8m23.680665261s) [noise=true , address=145.79.227.11:4732 , piece_id=NXZGC73AGULETYQ2EMVQ7XM36TNFNECSTUGTI3VMCPXJGGY2ALQA, seg_pos={0 0}, upload=533848]: Done uploading piece. err: <nil> Dec 31 00:37:23.879984451 (609h8m23.751067486s) [noise=true , address=145.79.227.41:4086 , piece_id=KCXGGAOKZNWFLLBV7I6P7LHVNM6DFRN5DITCURNMWBA2VE7CKVHA, seg_pos={0 0}, upload=533848]: Done uploading piece. err: <nil> Dec 31 00:37:23.982919423 (609h8m23.854002458s) [noise=true , address=77.90.31.13:4629 , piece_id=WAZYSDAN5MSE3C3UGJ5BQ7GDOPLPAOB7YQTHYUTIE7OZWRIVEARA, seg_pos={0 0}, upload=533848]: Done uploading piece. err: <nil> Dec 31 00:37:24.025039846 (609h8m23.896122881s) [noise=true , address=77.90.31.18:4805 , piece_id=QRFMMAPSLAYNXMN5JG2PKJ5DZUGPOXUSDBYRBGCYR34WT63APUOA, seg_pos={0 0}, upload=533848]: Done uploading piece. err: <nil> Dec 31 00:37:24.081686509 (609h8m23.952769695s) [noise=true , address=45.91.116.8:4045 , piece_id=RTVLMBX2SLOK23HMAWEA4G6WHEEZEBH6XQRKKRK6WD5GD6BDW7NA, seg_pos={0 0}, upload=533848]: Done uploading piece. err: <nil> Dec 31 00:37:24.116560253 (609h8m23.987643288s) [noise=true , address=45.91.116.5:4299 , piece_id=2GPR5IUAX2AHYXUSSLGXOOU6C3PYEPIY6P4B67GBTMIE5X4FFVHQ, seg_pos={0 0}, upload=533848]: Done uploading piece. err: <nil> Dec 31 00:37:24.152081901 (609h8m24.023164936s) [noise=true , address=45.91.116.10:4081 , piece_id=6GJLWPKDJL6TUXLQLGS62F7YG6PPZ42O3VQT6KQFNTTBOZVSFDGQ, seg_pos={0 0}, upload=533848]: Done uploading piece. err: <nil> Dec 31 00:37:24.167484239 (609h8m24.038567284s) [noise=true , address=77.90.31.14:4663 , piece_id=BEL6CMHHCPH6PJBFQBY2P22GI2DSHLU6C2FI3KREEKFXVESBJROQ, seg_pos={0 0}, upload=533848]: Done uploading piece. err: <nil> Dec 31 00:37:24.199729909 (609h8m24.070812944s) [noise=true , address=45.91.116.9:4262 , piece_id=6FNAPYOYWGGJYCTPXCMWI7L325YOIP6RDE7UZGJUH5XZICAA4CTQ, seg_pos={0 0}, upload=533848]: Done uploading piece. err: <nil> Dec 31 00:37:24.728312881 (609h8m24.599395926s) [noise=true , address=77.90.31.21:4878 , piece_id=T6MNCWRE4CB45ARESCQV3UCXA6FHSBVSC6SFJCRA4FOKL4QJ7Y2A, seg_pos={0 0}, upload=533848]: Done uploading piece. err: <nil> Dec 31 00:37:24.728331858 (609h8m24.599414893s) [seg_pos={0 0}, upload=533848]: Segment reached optimal threshold of 13 pieces. Dec 31 00:37:24.728520977 (609h8m24.599604013s) [noise=true , address=145.79.227.43:4157 , piece_id=42TVBZYTZ4O3AJ5FHBZMPBMWMEUIZDGY5NXYHAZ3TBKNHG6QCNRQ, seg_pos={0 0}, upload=533848]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZrzADk2mKG4R9tjxUKjQeJvb4WmfjLksGFyvstHRfYbBC7XNz): protocol: expected piece hash; context canceled; EOF Dec 31 00:37:24.728527470 (609h8m24.599610505s) [noise=true , address=145.79.227.43:4157 , piece_id=42TVBZYTZ4O3AJ5FHBZMPBMWMEUIZDGY5NXYHAZ3TBKNHG6QCNRQ, seg_pos={0 0}, upload=533848]: Long tail cancellation due to optimal threshold Dec 31 00:37:24.728572045 (609h8m24.59965508s) [noise=true , address=145.79.227.29:4942 , piece_id=3M3YMDPBFYAW2RZGTFMZLPVGDP7HY72DLHR2PALDI6FJFYDLHFAA, seg_pos={0 0}, upload=533848]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17o881pUMuG7K2dv83k7RkrJCrmM7kgUiWesgam4GEw21KVUn7): protocol: expected piece hash; context canceled; EOF Dec 31 00:37:24.728580772 (609h8m24.599663807s) [noise=true , address=145.79.227.29:4942 , piece_id=3M3YMDPBFYAW2RZGTFMZLPVGDP7HY72DLHR2PALDI6FJFYDLHFAA, seg_pos={0 0}, upload=533848]: Long tail cancellation due to optimal threshold Dec 31 00:37:24.728718213 (609h8m24.599801258s) [noise=true , address=45.91.116.6:4975 , piece_id=AC5RI6DJ3L5QPO22FF3J3KQW2UEQVC5QBD247E35JIMIIQ5OQP4A, seg_pos={0 0}, upload=533848]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Dec 31 00:37:24.728724195 (609h8m24.59980723s) [noise=true , address=45.91.116.6:4975 , piece_id=AC5RI6DJ3L5QPO22FF3J3KQW2UEQVC5QBD247E35JIMIIQ5OQP4A, seg_pos={0 0}, upload=533848]: Long tail cancellation due to optimal threshold Dec 31 00:37:24.728742089 (609h8m24.599825124s) [seg_pos={0 0}, upload=533848]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 00:37:24.728824065 (609h8m24.5999071s) [upload=533848]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 00:37:24.728828854 (609h8m24.599911889s) [upload=533848]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 00:37:25.041591605 (609h8m24.91267466s) [upload=533848]: Done uploading segments... Dec 31 00:37:25.041615270 (609h8m24.912698315s) [upload=533848]: Upload finished. err: <nil> Duration: 5.877211224s core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 4.831922105s trace: https://jaeger.dev.storj.io/trace/63d344eb9b98ed45 FAIL (2025-12-31 00:41:01 UTC) orbiter warning: operation: DeleteObject uplink: metaclient: context deadline exceeded Duration: 4.831943905s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 31.248784367s trace: https://jaeger.dev.storj.io/trace/67f6164cdd15ab81 FAIL (2025-12-31 00:40:28 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== 31 00:33:25.026367244 (609h4m24.897450289s) [noise=true , address=145.79.227.2:4128 , piece_id=DW37Q32FFAZ37KHMOII6XGCJVDO2XTKED5TLOKG65JULWTXJIQEA, seg_pos={0 0}, upload=533792]: Uploading piece... Dec 31 00:33:25.026878247 (609h4m24.897961292s) [noise=true , address=145.79.227.44:4217 , piece_id=GW44B36EHWVCRLEM6BA5MXC3ITRNCCJATW2RXSYTNC7K7WRO2IDQ, seg_pos={0 0}, upload=533792]: Uploading piece... Dec 31 00:33:25.027303136 (609h4m24.898386171s) [noise=true , address=45.91.116.13:4161 , piece_id=K4CTNQGHL5S2GOARW2F42REGZ4QVWA2RKUNULONYX3VPVHAGONUQ, seg_pos={0 0}, upload=533792]: Uploading piece... Dec 31 00:33:25.027901605 (609h4m24.89898465s) [noise=true , address=77.90.31.13:4632 , piece_id=5KUEGWEJV5NJE2EX3S5S2G7TS5TK23ZQXV3WPA4LLHO65QJRVW6Q, seg_pos={0 0}, upload=533792]: Uploading piece... Dec 31 00:33:25.028257373 (609h4m24.899340408s) [noise=true , address=145.79.227.30:4769 , piece_id=P2ANRKDKCOTOMCZZMDIH2YRVI366UOHWO3LXV7INEDB2GW35Y52A, seg_pos={0 0}, upload=533792]: Uploading piece... Dec 31 00:33:25.028623710 (609h4m24.899706745s) [noise=true , address=145.79.227.34:4930 , piece_id=L3AVRBWPDFTZANZKO4TGITCA3WHBECKEQ2BGMLFQJYN4CR7FSADA, seg_pos={0 0}, upload=533792]: Uploading piece... Dec 31 00:33:25.028952256 (609h4m24.900035291s) [noise=true , address=145.79.227.25:4630 , piece_id=5BVJWWAW2QUQABWWZCD4EW54ZVH3K5RJNM5EXJJUGXI33O6CC24A, seg_pos={0 0}, upload=533792]: Uploading piece... Dec 31 00:33:25.029314766 (609h4m24.900397801s) [noise=true , address=145.79.227.29:4971 , piece_id=EIY6YRG5ZRMEW7IWE5X25RCTGNCU3HQ27O7Q434OAZ4EI335HU4A, seg_pos={0 0}, upload=533792]: Uploading piece... Dec 31 00:33:25.029659533 (609h4m24.900742578s) [noise=true , address=145.79.227.6:4283 , piece_id=MXSG7TVTQBR2AFONVXYWQB4DBAQEEXW226FDC6SSVG5TTWGPAWFQ, seg_pos={0 0}, upload=533792]: Uploading piece... Dec 31 00:33:25.030576118 (609h4m24.901659153s) [noise=true , address=77.90.31.18:4835 , piece_id=GRMJTN6ZDDYITHJWX5UGIAGLCOW6AKADHTK6UCVGRB437TUTDFGA, seg_pos={0 0}, upload=533792]: Uploading piece... Dec 31 00:33:25.031413452 (609h4m24.902496487s) [noise=true , address=64.49.13.155:4913 , piece_id=E4WJCMYJVURPHENWEWR46KOAUAOZU7FW7WLC53DAU44BE7R3CHMQ, seg_pos={0 0}, upload=533792]: Uploading piece... Dec 31 00:33:25.031987204 (609h4m24.903070239s) [noise=true , address=145.79.227.49:4370 , piece_id=52YCNJY7BLJ3N3HDWBZCM7PP2ZPHHBS4GV4QS4WM2AWSLYKB5BBA, seg_pos={0 0}, upload=533792]: Uploading piece... Dec 31 00:33:25.032849947 (609h4m24.903932993s) [noise=true , address=145.79.227.4:4189 , piece_id=ZZF4DSIPQBWKBD45UR2J5P75U5VVMTVB54NEX5QKZV4K2WJZ43BQ, seg_pos={0 0}, upload=533792]: Uploading piece... Dec 31 00:33:25.034222320 (609h4m24.905305355s) [noise=true , address=145.79.227.49:4353 , piece_id=MVTAW6QPJY2LRNEGISHNDP7AC5ZIIUR5N5BBZRFO5I6VYJU7UR3Q, seg_pos={0 0}, upload=533792]: Uploading piece... Dec 31 00:33:25.035359376 (609h4m24.906442411s) [noise=true , address=145.79.227.35:4494 , piece_id=EY2OJLQIF4CH7DQWJWXSBJMQI37NAXSBLQP4ZPMU3IBRVLWJVYWA, seg_pos={0 0}, upload=533792]: Uploading piece... Dec 31 00:33:25.035852344 (609h4m24.906935379s) [noise=true , address=145.79.227.33:4873 , piece_id=2DSEGSC5UWL76NEXSE7WXOGSQNRNXALAINY52TLQHB57LTS2R6ZQ, seg_pos={0 0}, upload=533792]: Uploading piece... Dec 31 00:33:26.491429069 (609h4m26.362512104s) [upload=533792]: Next returned nil segment Dec 31 00:33:26.491441904 (609h4m26.362524939s) [upload=533792]: Waiting for error group managing segments... Dec 31 00:33:26.876786140 (609h4m26.747869175s) [noise=true , address=145.79.227.6:4283 , piece_id=MXSG7TVTQBR2AFONVXYWQB4DBAQEEXW226FDC6SSVG5TTWGPAWFQ, seg_pos={0 0}, upload=533792]: Done uploading piece. err: <nil> Dec 31 00:33:26.895923271 (609h4m26.767006316s) [noise=true , address=145.79.227.35:4494 , piece_id=EY2OJLQIF4CH7DQWJWXSBJMQI37NAXSBLQP4ZPMU3IBRVLWJVYWA, seg_pos={0 0}, upload=533792]: Done uploading piece. err: <nil> Dec 31 00:33:26.903106888 (609h4m26.774189923s) [noise=true , address=145.79.227.33:4873 , piece_id=2DSEGSC5UWL76NEXSE7WXOGSQNRNXALAINY52TLQHB57LTS2R6ZQ, seg_pos={0 0}, upload=533792]: Done uploading piece. err: <nil> Dec 31 00:33:26.912484285 (609h4m26.78356733s) [noise=true , address=145.79.227.29:4971 , piece_id=EIY6YRG5ZRMEW7IWE5X25RCTGNCU3HQ27O7Q434OAZ4EI335HU4A, seg_pos={0 0}, upload=533792]: Done uploading piece. err: <nil> Dec 31 00:33:27.004280888 (609h4m26.875363933s) [noise=true , address=145.79.227.21:4495 , piece_id=YWETKWGBGTY3LALK4DXWSKUTM2TFEYB4N5A3KNKW346V3F5XRN2Q, seg_pos={0 0}, upload=533792]: Done uploading piece. err: <nil> Dec 31 00:33:27.118759862 (609h4m26.989842897s) [noise=true , address=77.90.31.13:4632 , piece_id=5KUEGWEJV5NJE2EX3S5S2G7TS5TK23ZQXV3WPA4LLHO65QJRVW6Q, seg_pos={0 0}, upload=533792]: Done uploading piece. err: <nil> Dec 31 00:33:27.225324635 (609h4m27.09640767s) [noise=true , address=45.91.116.13:4161 , piece_id=K4CTNQGHL5S2GOARW2F42REGZ4QVWA2RKUNULONYX3VPVHAGONUQ, seg_pos={0 0}, upload=533792]: Done uploading piece. err: <nil> Dec 31 00:33:27.347765488 (609h4m27.218848533s) [noise=true , address=45.91.116.12:4248 , piece_id=QXKIUKBDX2GSJ2EQPALFCLPPLNWBPL3GRAEHLPJNSWUDIMU7WHYA, seg_pos={0 0}, upload=533792]: Done uploading piece. err: <nil> Dec 31 00:33:27.789508217 (609h4m27.660591252s) [noise=true , address=145.79.227.44:4217 , piece_id=GW44B36EHWVCRLEM6BA5MXC3ITRNCCJATW2RXSYTNC7K7WRO2IDQ, seg_pos={0 0}, upload=533792]: Done uploading piece. err: <nil> Dec 31 00:33:27.821042021 (609h4m27.692125056s) [noise=true , address=145.79.227.49:4353 , piece_id=MVTAW6QPJY2LRNEGISHNDP7AC5ZIIUR5N5BBZRFO5I6VYJU7UR3Q, seg_pos={0 0}, upload=533792]: Done uploading piece. err: <nil> Dec 31 00:33:27.828648224 (609h4m27.699731259s) [noise=true , address=145.79.227.49:4370 , piece_id=52YCNJY7BLJ3N3HDWBZCM7PP2ZPHHBS4GV4QS4WM2AWSLYKB5BBA, seg_pos={0 0}, upload=533792]: Done uploading piece. err: <nil> Dec 31 00:33:27.840271497 (609h4m27.711354542s) [noise=true , address=145.79.227.30:4769 , piece_id=P2ANRKDKCOTOMCZZMDIH2YRVI366UOHWO3LXV7INEDB2GW35Y52A, seg_pos={0 0}, upload=533792]: Done uploading piece. err: <nil> Dec 31 00:33:27.844064569 (609h4m27.715147614s) [noise=true , address=145.79.227.34:4930 , piece_id=L3AVRBWPDFTZANZKO4TGITCA3WHBECKEQ2BGMLFQJYN4CR7FSADA, seg_pos={0 0}, upload=533792]: Done uploading piece. err: <nil> Dec 31 00:33:27.867248213 (609h4m27.738331248s) [noise=true , address=77.90.31.18:4835 , piece_id=GRMJTN6ZDDYITHJWX5UGIAGLCOW6AKADHTK6UCVGRB437TUTDFGA, seg_pos={0 0}, upload=533792]: Done uploading piece. err: <nil> Dec 31 00:33:27.867270586 (609h4m27.738353621s) [seg_pos={0 0}, upload=533792]: Segment reached optimal threshold of 14 pieces. Dec 31 00:33:27.867476859 (609h4m27.738559904s) [noise=true , address=64.49.13.155:4913 , piece_id=E4WJCMYJVURPHENWEWR46KOAUAOZU7FW7WLC53DAU44BE7R3CHMQ, seg_pos={0 0}, upload=533792]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oyYUnWDyFo1npaKpBxMG5K31oUzJnTpcU1XHFmKux3MCVUPZV): protocol: expected piece hash; context canceled; EOF Dec 31 00:33:27.867485555 (609h4m27.73856859s) [noise=true , address=64.49.13.155:4913 , piece_id=E4WJCMYJVURPHENWEWR46KOAUAOZU7FW7WLC53DAU44BE7R3CHMQ, seg_pos={0 0}, upload=533792]: Long tail cancellation due to optimal threshold Dec 31 00:33:27.867516204 (609h4m27.738599249s) [noise=true , address=145.79.227.25:4630 , piece_id=5BVJWWAW2QUQABWWZCD4EW54ZVH3K5RJNM5EXJJUGXI33O6CC24A, seg_pos={0 0}, upload=533792]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12aETKgTzKw8fyYqbeDA2JzfxZnWn12xUPYK1eRt6TvBPPdRJDp): protocol: expected piece hash; context canceled; EOF Dec 31 00:33:27.867526553 (609h4m27.738609588s) [noise=true , address=145.79.227.25:4630 , piece_id=5BVJWWAW2QUQABWWZCD4EW54ZVH3K5RJNM5EXJJUGXI33O6CC24A, seg_pos={0 0}, upload=533792]: Long tail cancellation due to optimal threshold Dec 31 00:33:27.867714432 (609h4m27.738797477s) [noise=true , address=145.79.227.4:4189 , piece_id=ZZF4DSIPQBWKBD45UR2J5P75U5VVMTVB54NEX5QKZV4K2WJZ43BQ, seg_pos={0 0}, upload=533792]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jH3757i2F3q18XSpoXznsysZPaXmEnqynpLPDupgRfxJrTgKv): protocol: write tcp 188.166.225.170:57528->145.79.227.4:4189: use of closed network connection; write tcp 188.166.225.170:57528->145.79.227.4:4189: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57528->145.79.227.4:4189: use of closed network connection Dec 31 00:33:27.867723689 (609h4m27.738806724s) [noise=true , address=145.79.227.4:4189 , piece_id=ZZF4DSIPQBWKBD45UR2J5P75U5VVMTVB54NEX5QKZV4K2WJZ43BQ, seg_pos={0 0}, upload=533792]: Long tail cancellation due to optimal threshold Dec 31 00:33:27.867744699 (609h4m27.738827734s) [noise=true , address=145.79.227.2:4128 , piece_id=DW37Q32FFAZ37KHMOII6XGCJVDO2XTKED5TLOKG65JULWTXJIQEA, seg_pos={0 0}, upload=533792]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1wABkAtAjoYkKyNoAiaVCxSvY6WgvHvAYFPCjpNojtpiYiGtDS): protocol: expected piece hash; context canceled; EOF Dec 31 00:33:27.867753616 (609h4m27.738836651s) [noise=true , address=145.79.227.2:4128 , piece_id=DW37Q32FFAZ37KHMOII6XGCJVDO2XTKED5TLOKG65JULWTXJIQEA, seg_pos={0 0}, upload=533792]: Long tail cancellation due to optimal threshold Dec 31 00:33:27.867777462 (609h4m27.738860507s) [seg_pos={0 0}, upload=533792]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 31 00:33:27.867853687 (609h4m27.738936722s) [upload=533792]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 00:33:27.867859989 (609h4m27.738943024s) [upload=533792]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 00:33:28.167335511 (609h4m28.038418546s) [upload=533792]: Done uploading segments... Dec 31 00:33:28.167355899 (609h4m28.038438934s) [upload=533792]: Upload finished. err: <nil> Duration: 31.248800329s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [.............X...X...........X........X..X....XX.XXX....X...] duration: 15.963103386s trace: https://jaeger.dev.storj.io/trace/4804197dbd063942 FAIL (2025-12-31 00:40:19 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, Piece ID: QRFMMAPSLAYNXMN5JG2PKJ5DZUGPOXUSDBYRBGCYR34WT63APUOA) context deadline exceeded; piecestore: (Node ID: 12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f, Piece ID: WAZYSDAN5MSE3C3UGJ5BQ7GDOPLPAOB7YQTHYUTIE7OZWRIVEARA) context deadline exceeded ======LOGS====== Dec 31 00:37:21.551769648 (609h8m21.422852683s) [upload=533848]: Starting upload Dec 31 00:37:21.551853758 (609h8m21.422936803s) [upload=533848]: Done starting upload Dec 31 00:37:21.551965832 (609h8m21.423048887s) [upload=533848]: Uploading segments... Dec 31 00:37:21.552177856 (609h8m21.423260901s) [upload=533848]: Got next segment. Inline: false Dec 31 00:37:21.552220857 (609h8m21.423303902s) [upload=533848]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 00:37:21.552226658 (609h8m21.423309703s) [upload=533848]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 00:37:21.804536710 (609h8m21.675619755s) [seg_pos={0 0}, upload=533848]: Begin upload segment... Dec 31 00:37:21.804586926 (609h8m21.675669961s) [seg_pos={0 0}, upload=533848]: Done begin upload segment. Dec 31 00:37:21.804611863 (609h8m21.675694908s) [noise=true , address=77.90.31.18:4805 , piece_id=QRFMMAPSLAYNXMN5JG2PKJ5DZUGPOXUSDBYRBGCYR34WT63APUOA, seg_pos={0 0}, upload=533848]: Uploading piece... Dec 31 00:37:21.804621622 (609h8m21.675704657s) [noise=true , address=77.90.31.21:4878 , piece_id=T6MNCWRE4CB45ARESCQV3UCXA6FHSBVSC6SFJCRA4FOKL4QJ7Y2A, seg_pos={0 0}, upload=533848]: Uploading piece... Dec 31 00:37:21.805095093 (609h8m21.676178138s) [noise=true , address=145.79.227.11:4732 , piece_id=NXZGC73AGULETYQ2EMVQ7XM36TNFNECSTUGTI3VMCPXJGGY2ALQA, seg_pos={0 0}, upload=533848]: Uploading piece... Dec 31 00:37:21.805232335 (609h8m21.67631537s) [noise=true , address=45.91.116.5:4299 , piece_id=2GPR5IUAX2AHYXUSSLGXOOU6C3PYEPIY6P4B67GBTMIE5X4FFVHQ, seg_pos={0 0}, upload=533848]: Uploading piece... Dec 31 00:37:21.805605015 (609h8m21.67668806s) [noise=true , address=145.79.227.41:4086 , piece_id=KCXGGAOKZNWFLLBV7I6P7LHVNM6DFRN5DITCURNMWBA2VE7CKVHA, seg_pos={0 0}, upload=533848]: Uploading piece... Dec 31 00:37:21.805935163 (609h8m21.677018198s) [noise=true , address=45.91.116.6:4975 , piece_id=AC5RI6DJ3L5QPO22FF3J3KQW2UEQVC5QBD247E35JIMIIQ5OQP4A, seg_pos={0 0}, upload=533848]: Uploading piece... Dec 31 00:37:21.805997943 (609h8m21.677080988s) [noise=true , address=45.91.116.8:4045 , piece_id=RTVLMBX2SLOK23HMAWEA4G6WHEEZEBH6XQRKKRK6WD5GD6BDW7NA, seg_pos={0 0}, upload=533848]: Uploading piece... Dec 31 00:37:21.806193325 (609h8m21.67727636s) [noise=true , address=145.79.227.31:4797 , piece_id=IZMS3LM7IG3RBXIMQTRDCINIQBQ4MNQM4XXDSJUTEJBBVTMSEZEA, seg_pos={0 0}, upload=533848]: Uploading piece... Dec 31 00:37:21.806390720 (609h8m21.677473765s) [noise=true , address=45.91.116.10:4081 , piece_id=6GJLWPKDJL6TUXLQLGS62F7YG6PPZ42O3VQT6KQFNTTBOZVSFDGQ, seg_pos={0 0}, upload=533848]: Uploading piece... Dec 31 00:37:21.806476133 (609h8m21.677559178s) [noise=true , address=145.79.227.43:4157 , piece_id=42TVBZYTZ4O3AJ5FHBZMPBMWMEUIZDGY5NXYHAZ3TBKNHG6QCNRQ, seg_pos={0 0}, upload=533848]: Uploading piece... Dec 31 00:37:21.806866076 (609h8m21.677949121s) [noise=true , address=145.79.227.19:4079 , piece_id=W2AGJA3O2XARCZEI5USBHZE6EIWIDAFU24WKPCUHQII65YQ2VAKA, seg_pos={0 0}, upload=533848]: Uploading piece... Dec 31 00:37:21.806964183 (609h8m21.678047228s) [noise=true , address=77.90.31.13:4629 , piece_id=WAZYSDAN5MSE3C3UGJ5BQ7GDOPLPAOB7YQTHYUTIE7OZWRIVEARA, seg_pos={0 0}, upload=533848]: Uploading piece... Dec 31 00:37:21.807173682 (609h8m21.678256717s) [noise=true , address=145.79.227.10:4402 , piece_id=UJXD3JBNBCKBK2H4OAW4KBCFXMZMP3MHDKYZOYFYSKCAJDWIU4YQ, seg_pos={0 0}, upload=533848]: Uploading piece... Dec 31 00:37:21.807465367 (609h8m21.678548402s) [noise=true , address=45.91.116.9:4262 , piece_id=6FNAPYOYWGGJYCTPXCMWI7L325YOIP6RDE7UZGJUH5XZICAA4CTQ, seg_pos={0 0}, upload=533848]: Uploading piece... Dec 31 00:37:21.807912939 (609h8m21.678995984s) [noise=true , address=77.90.31.14:4663 , piece_id=BEL6CMHHCPH6PJBFQBY2P22GI2DSHLU6C2FI3KREEKFXVESBJROQ, seg_pos={0 0}, upload=533848]: Uploading piece... Dec 31 00:37:21.808755323 (609h8m21.679838368s) [noise=true , address=145.79.227.29:4942 , piece_id=3M3YMDPBFYAW2RZGTFMZLPVGDP7HY72DLHR2PALDI6FJFYDLHFAA, seg_pos={0 0}, upload=533848]: Uploading piece... Dec 31 00:37:23.506877615 (609h8m23.37796065s) [upload=533848]: Next returned nil segment Dec 31 00:37:23.506893356 (609h8m23.377976401s) [upload=533848]: Waiting for error group managing segments... Dec 31 00:37:23.787196661 (609h8m23.658279696s) [noise=true , address=145.79.227.19:4079 , piece_id=W2AGJA3O2XARCZEI5USBHZE6EIWIDAFU24WKPCUHQII65YQ2VAKA, seg_pos={0 0}, upload=533848]: Done uploading piece. err: <nil> Dec 31 00:37:23.794707001 (609h8m23.665790046s) [noise=true , address=145.79.227.31:4797 , piece_id=IZMS3LM7IG3RBXIMQTRDCINIQBQ4MNQM4XXDSJUTEJBBVTMSEZEA, seg_pos={0 0}, upload=533848]: Done uploading piece. err: <nil> Dec 31 00:37:23.802772670 (609h8m23.673855705s) [noise=true , address=145.79.227.10:4402 , piece_id=UJXD3JBNBCKBK2H4OAW4KBCFXMZMP3MHDKYZOYFYSKCAJDWIU4YQ, seg_pos={0 0}, upload=533848]: Done uploading piece. err: <nil> Dec 31 00:37:23.809582226 (609h8m23.680665261s) [noise=true , address=145.79.227.11:4732 , piece_id=NXZGC73AGULETYQ2EMVQ7XM36TNFNECSTUGTI3VMCPXJGGY2ALQA, seg_pos={0 0}, upload=533848]: Done uploading piece. err: <nil> Dec 31 00:37:23.879984451 (609h8m23.751067486s) [noise=true , address=145.79.227.41:4086 , piece_id=KCXGGAOKZNWFLLBV7I6P7LHVNM6DFRN5DITCURNMWBA2VE7CKVHA, seg_pos={0 0}, upload=533848]: Done uploading piece. err: <nil> Dec 31 00:37:23.982919423 (609h8m23.854002458s) [noise=true , address=77.90.31.13:4629 , piece_id=WAZYSDAN5MSE3C3UGJ5BQ7GDOPLPAOB7YQTHYUTIE7OZWRIVEARA, seg_pos={0 0}, upload=533848]: Done uploading piece. err: <nil> Dec 31 00:37:24.025039846 (609h8m23.896122881s) [noise=true , address=77.90.31.18:4805 , piece_id=QRFMMAPSLAYNXMN5JG2PKJ5DZUGPOXUSDBYRBGCYR34WT63APUOA, seg_pos={0 0}, upload=533848]: Done uploading piece. err: <nil> Dec 31 00:37:24.081686509 (609h8m23.952769695s) [noise=true , address=45.91.116.8:4045 , piece_id=RTVLMBX2SLOK23HMAWEA4G6WHEEZEBH6XQRKKRK6WD5GD6BDW7NA, seg_pos={0 0}, upload=533848]: Done uploading piece. err: <nil> Dec 31 00:37:24.116560253 (609h8m23.987643288s) [noise=true , address=45.91.116.5:4299 , piece_id=2GPR5IUAX2AHYXUSSLGXOOU6C3PYEPIY6P4B67GBTMIE5X4FFVHQ, seg_pos={0 0}, upload=533848]: Done uploading piece. err: <nil> Dec 31 00:37:24.152081901 (609h8m24.023164936s) [noise=true , address=45.91.116.10:4081 , piece_id=6GJLWPKDJL6TUXLQLGS62F7YG6PPZ42O3VQT6KQFNTTBOZVSFDGQ, seg_pos={0 0}, upload=533848]: Done uploading piece. err: <nil> Dec 31 00:37:24.167484239 (609h8m24.038567284s) [noise=true , address=77.90.31.14:4663 , piece_id=BEL6CMHHCPH6PJBFQBY2P22GI2DSHLU6C2FI3KREEKFXVESBJROQ, seg_pos={0 0}, upload=533848]: Done uploading piece. err: <nil> Dec 31 00:37:24.199729909 (609h8m24.070812944s) [noise=true , address=45.91.116.9:4262 , piece_id=6FNAPYOYWGGJYCTPXCMWI7L325YOIP6RDE7UZGJUH5XZICAA4CTQ, seg_pos={0 0}, upload=533848]: Done uploading piece. err: <nil> Dec 31 00:37:24.728312881 (609h8m24.599395926s) [noise=true , address=77.90.31.21:4878 , piece_id=T6MNCWRE4CB45ARESCQV3UCXA6FHSBVSC6SFJCRA4FOKL4QJ7Y2A, seg_pos={0 0}, upload=533848]: Done uploading piece. err: <nil> Dec 31 00:37:24.728331858 (609h8m24.599414893s) [seg_pos={0 0}, upload=533848]: Segment reached optimal threshold of 13 pieces. Dec 31 00:37:24.728520977 (609h8m24.599604013s) [noise=true , address=145.79.227.43:4157 , piece_id=42TVBZYTZ4O3AJ5FHBZMPBMWMEUIZDGY5NXYHAZ3TBKNHG6QCNRQ, seg_pos={0 0}, upload=533848]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZrzADk2mKG4R9tjxUKjQeJvb4WmfjLksGFyvstHRfYbBC7XNz): protocol: expected piece hash; context canceled; EOF Dec 31 00:37:24.728527470 (609h8m24.599610505s) [noise=true , address=145.79.227.43:4157 , piece_id=42TVBZYTZ4O3AJ5FHBZMPBMWMEUIZDGY5NXYHAZ3TBKNHG6QCNRQ, seg_pos={0 0}, upload=533848]: Long tail cancellation due to optimal threshold Dec 31 00:37:24.728572045 (609h8m24.59965508s) [noise=true , address=145.79.227.29:4942 , piece_id=3M3YMDPBFYAW2RZGTFMZLPVGDP7HY72DLHR2PALDI6FJFYDLHFAA, seg_pos={0 0}, upload=533848]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17o881pUMuG7K2dv83k7RkrJCrmM7kgUiWesgam4GEw21KVUn7): protocol: expected piece hash; context canceled; EOF Dec 31 00:37:24.728580772 (609h8m24.599663807s) [noise=true , address=145.79.227.29:4942 , piece_id=3M3YMDPBFYAW2RZGTFMZLPVGDP7HY72DLHR2PALDI6FJFYDLHFAA, seg_pos={0 0}, upload=533848]: Long tail cancellation due to optimal threshold Dec 31 00:37:24.728718213 (609h8m24.599801258s) [noise=true , address=45.91.116.6:4975 , piece_id=AC5RI6DJ3L5QPO22FF3J3KQW2UEQVC5QBD247E35JIMIIQ5OQP4A, seg_pos={0 0}, upload=533848]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Dec 31 00:37:24.728724195 (609h8m24.59980723s) [noise=true , address=45.91.116.6:4975 , piece_id=AC5RI6DJ3L5QPO22FF3J3KQW2UEQVC5QBD247E35JIMIIQ5OQP4A, seg_pos={0 0}, upload=533848]: Long tail cancellation due to optimal threshold Dec 31 00:37:24.728742089 (609h8m24.599825124s) [seg_pos={0 0}, upload=533848]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 00:37:24.728824065 (609h8m24.5999071s) [upload=533848]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 00:37:24.728828854 (609h8m24.599911889s) [upload=533848]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 00:37:25.041591605 (609h8m24.91267466s) [upload=533848]: Done uploading segments... Dec 31 00:37:25.041615270 (609h8m24.912698315s) [upload=533848]: Upload finished. err: <nil> Duration: 15.963133312s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 32.10872328s trace: https://jaeger.dev.storj.io/trace/67f6164cdd15ab81 FAIL (2025-12-31 00:39:56 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== 31 00:33:25.026367244 (609h4m24.897450289s) [noise=true , address=145.79.227.2:4128 , piece_id=DW37Q32FFAZ37KHMOII6XGCJVDO2XTKED5TLOKG65JULWTXJIQEA, seg_pos={0 0}, upload=533792]: Uploading piece... Dec 31 00:33:25.026878247 (609h4m24.897961292s) [noise=true , address=145.79.227.44:4217 , piece_id=GW44B36EHWVCRLEM6BA5MXC3ITRNCCJATW2RXSYTNC7K7WRO2IDQ, seg_pos={0 0}, upload=533792]: Uploading piece... Dec 31 00:33:25.027303136 (609h4m24.898386171s) [noise=true , address=45.91.116.13:4161 , piece_id=K4CTNQGHL5S2GOARW2F42REGZ4QVWA2RKUNULONYX3VPVHAGONUQ, seg_pos={0 0}, upload=533792]: Uploading piece... Dec 31 00:33:25.027901605 (609h4m24.89898465s) [noise=true , address=77.90.31.13:4632 , piece_id=5KUEGWEJV5NJE2EX3S5S2G7TS5TK23ZQXV3WPA4LLHO65QJRVW6Q, seg_pos={0 0}, upload=533792]: Uploading piece... Dec 31 00:33:25.028257373 (609h4m24.899340408s) [noise=true , address=145.79.227.30:4769 , piece_id=P2ANRKDKCOTOMCZZMDIH2YRVI366UOHWO3LXV7INEDB2GW35Y52A, seg_pos={0 0}, upload=533792]: Uploading piece... Dec 31 00:33:25.028623710 (609h4m24.899706745s) [noise=true , address=145.79.227.34:4930 , piece_id=L3AVRBWPDFTZANZKO4TGITCA3WHBECKEQ2BGMLFQJYN4CR7FSADA, seg_pos={0 0}, upload=533792]: Uploading piece... Dec 31 00:33:25.028952256 (609h4m24.900035291s) [noise=true , address=145.79.227.25:4630 , piece_id=5BVJWWAW2QUQABWWZCD4EW54ZVH3K5RJNM5EXJJUGXI33O6CC24A, seg_pos={0 0}, upload=533792]: Uploading piece... Dec 31 00:33:25.029314766 (609h4m24.900397801s) [noise=true , address=145.79.227.29:4971 , piece_id=EIY6YRG5ZRMEW7IWE5X25RCTGNCU3HQ27O7Q434OAZ4EI335HU4A, seg_pos={0 0}, upload=533792]: Uploading piece... Dec 31 00:33:25.029659533 (609h4m24.900742578s) [noise=true , address=145.79.227.6:4283 , piece_id=MXSG7TVTQBR2AFONVXYWQB4DBAQEEXW226FDC6SSVG5TTWGPAWFQ, seg_pos={0 0}, upload=533792]: Uploading piece... Dec 31 00:33:25.030576118 (609h4m24.901659153s) [noise=true , address=77.90.31.18:4835 , piece_id=GRMJTN6ZDDYITHJWX5UGIAGLCOW6AKADHTK6UCVGRB437TUTDFGA, seg_pos={0 0}, upload=533792]: Uploading piece... Dec 31 00:33:25.031413452 (609h4m24.902496487s) [noise=true , address=64.49.13.155:4913 , piece_id=E4WJCMYJVURPHENWEWR46KOAUAOZU7FW7WLC53DAU44BE7R3CHMQ, seg_pos={0 0}, upload=533792]: Uploading piece... Dec 31 00:33:25.031987204 (609h4m24.903070239s) [noise=true , address=145.79.227.49:4370 , piece_id=52YCNJY7BLJ3N3HDWBZCM7PP2ZPHHBS4GV4QS4WM2AWSLYKB5BBA, seg_pos={0 0}, upload=533792]: Uploading piece... Dec 31 00:33:25.032849947 (609h4m24.903932993s) [noise=true , address=145.79.227.4:4189 , piece_id=ZZF4DSIPQBWKBD45UR2J5P75U5VVMTVB54NEX5QKZV4K2WJZ43BQ, seg_pos={0 0}, upload=533792]: Uploading piece... Dec 31 00:33:25.034222320 (609h4m24.905305355s) [noise=true , address=145.79.227.49:4353 , piece_id=MVTAW6QPJY2LRNEGISHNDP7AC5ZIIUR5N5BBZRFO5I6VYJU7UR3Q, seg_pos={0 0}, upload=533792]: Uploading piece... Dec 31 00:33:25.035359376 (609h4m24.906442411s) [noise=true , address=145.79.227.35:4494 , piece_id=EY2OJLQIF4CH7DQWJWXSBJMQI37NAXSBLQP4ZPMU3IBRVLWJVYWA, seg_pos={0 0}, upload=533792]: Uploading piece... Dec 31 00:33:25.035852344 (609h4m24.906935379s) [noise=true , address=145.79.227.33:4873 , piece_id=2DSEGSC5UWL76NEXSE7WXOGSQNRNXALAINY52TLQHB57LTS2R6ZQ, seg_pos={0 0}, upload=533792]: Uploading piece... Dec 31 00:33:26.491429069 (609h4m26.362512104s) [upload=533792]: Next returned nil segment Dec 31 00:33:26.491441904 (609h4m26.362524939s) [upload=533792]: Waiting for error group managing segments... Dec 31 00:33:26.876786140 (609h4m26.747869175s) [noise=true , address=145.79.227.6:4283 , piece_id=MXSG7TVTQBR2AFONVXYWQB4DBAQEEXW226FDC6SSVG5TTWGPAWFQ, seg_pos={0 0}, upload=533792]: Done uploading piece. err: <nil> Dec 31 00:33:26.895923271 (609h4m26.767006316s) [noise=true , address=145.79.227.35:4494 , piece_id=EY2OJLQIF4CH7DQWJWXSBJMQI37NAXSBLQP4ZPMU3IBRVLWJVYWA, seg_pos={0 0}, upload=533792]: Done uploading piece. err: <nil> Dec 31 00:33:26.903106888 (609h4m26.774189923s) [noise=true , address=145.79.227.33:4873 , piece_id=2DSEGSC5UWL76NEXSE7WXOGSQNRNXALAINY52TLQHB57LTS2R6ZQ, seg_pos={0 0}, upload=533792]: Done uploading piece. err: <nil> Dec 31 00:33:26.912484285 (609h4m26.78356733s) [noise=true , address=145.79.227.29:4971 , piece_id=EIY6YRG5ZRMEW7IWE5X25RCTGNCU3HQ27O7Q434OAZ4EI335HU4A, seg_pos={0 0}, upload=533792]: Done uploading piece. err: <nil> Dec 31 00:33:27.004280888 (609h4m26.875363933s) [noise=true , address=145.79.227.21:4495 , piece_id=YWETKWGBGTY3LALK4DXWSKUTM2TFEYB4N5A3KNKW346V3F5XRN2Q, seg_pos={0 0}, upload=533792]: Done uploading piece. err: <nil> Dec 31 00:33:27.118759862 (609h4m26.989842897s) [noise=true , address=77.90.31.13:4632 , piece_id=5KUEGWEJV5NJE2EX3S5S2G7TS5TK23ZQXV3WPA4LLHO65QJRVW6Q, seg_pos={0 0}, upload=533792]: Done uploading piece. err: <nil> Dec 31 00:33:27.225324635 (609h4m27.09640767s) [noise=true , address=45.91.116.13:4161 , piece_id=K4CTNQGHL5S2GOARW2F42REGZ4QVWA2RKUNULONYX3VPVHAGONUQ, seg_pos={0 0}, upload=533792]: Done uploading piece. err: <nil> Dec 31 00:33:27.347765488 (609h4m27.218848533s) [noise=true , address=45.91.116.12:4248 , piece_id=QXKIUKBDX2GSJ2EQPALFCLPPLNWBPL3GRAEHLPJNSWUDIMU7WHYA, seg_pos={0 0}, upload=533792]: Done uploading piece. err: <nil> Dec 31 00:33:27.789508217 (609h4m27.660591252s) [noise=true , address=145.79.227.44:4217 , piece_id=GW44B36EHWVCRLEM6BA5MXC3ITRNCCJATW2RXSYTNC7K7WRO2IDQ, seg_pos={0 0}, upload=533792]: Done uploading piece. err: <nil> Dec 31 00:33:27.821042021 (609h4m27.692125056s) [noise=true , address=145.79.227.49:4353 , piece_id=MVTAW6QPJY2LRNEGISHNDP7AC5ZIIUR5N5BBZRFO5I6VYJU7UR3Q, seg_pos={0 0}, upload=533792]: Done uploading piece. err: <nil> Dec 31 00:33:27.828648224 (609h4m27.699731259s) [noise=true , address=145.79.227.49:4370 , piece_id=52YCNJY7BLJ3N3HDWBZCM7PP2ZPHHBS4GV4QS4WM2AWSLYKB5BBA, seg_pos={0 0}, upload=533792]: Done uploading piece. err: <nil> Dec 31 00:33:27.840271497 (609h4m27.711354542s) [noise=true , address=145.79.227.30:4769 , piece_id=P2ANRKDKCOTOMCZZMDIH2YRVI366UOHWO3LXV7INEDB2GW35Y52A, seg_pos={0 0}, upload=533792]: Done uploading piece. err: <nil> Dec 31 00:33:27.844064569 (609h4m27.715147614s) [noise=true , address=145.79.227.34:4930 , piece_id=L3AVRBWPDFTZANZKO4TGITCA3WHBECKEQ2BGMLFQJYN4CR7FSADA, seg_pos={0 0}, upload=533792]: Done uploading piece. err: <nil> Dec 31 00:33:27.867248213 (609h4m27.738331248s) [noise=true , address=77.90.31.18:4835 , piece_id=GRMJTN6ZDDYITHJWX5UGIAGLCOW6AKADHTK6UCVGRB437TUTDFGA, seg_pos={0 0}, upload=533792]: Done uploading piece. err: <nil> Dec 31 00:33:27.867270586 (609h4m27.738353621s) [seg_pos={0 0}, upload=533792]: Segment reached optimal threshold of 14 pieces. Dec 31 00:33:27.867476859 (609h4m27.738559904s) [noise=true , address=64.49.13.155:4913 , piece_id=E4WJCMYJVURPHENWEWR46KOAUAOZU7FW7WLC53DAU44BE7R3CHMQ, seg_pos={0 0}, upload=533792]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oyYUnWDyFo1npaKpBxMG5K31oUzJnTpcU1XHFmKux3MCVUPZV): protocol: expected piece hash; context canceled; EOF Dec 31 00:33:27.867485555 (609h4m27.73856859s) [noise=true , address=64.49.13.155:4913 , piece_id=E4WJCMYJVURPHENWEWR46KOAUAOZU7FW7WLC53DAU44BE7R3CHMQ, seg_pos={0 0}, upload=533792]: Long tail cancellation due to optimal threshold Dec 31 00:33:27.867516204 (609h4m27.738599249s) [noise=true , address=145.79.227.25:4630 , piece_id=5BVJWWAW2QUQABWWZCD4EW54ZVH3K5RJNM5EXJJUGXI33O6CC24A, seg_pos={0 0}, upload=533792]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12aETKgTzKw8fyYqbeDA2JzfxZnWn12xUPYK1eRt6TvBPPdRJDp): protocol: expected piece hash; context canceled; EOF Dec 31 00:33:27.867526553 (609h4m27.738609588s) [noise=true , address=145.79.227.25:4630 , piece_id=5BVJWWAW2QUQABWWZCD4EW54ZVH3K5RJNM5EXJJUGXI33O6CC24A, seg_pos={0 0}, upload=533792]: Long tail cancellation due to optimal threshold Dec 31 00:33:27.867714432 (609h4m27.738797477s) [noise=true , address=145.79.227.4:4189 , piece_id=ZZF4DSIPQBWKBD45UR2J5P75U5VVMTVB54NEX5QKZV4K2WJZ43BQ, seg_pos={0 0}, upload=533792]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jH3757i2F3q18XSpoXznsysZPaXmEnqynpLPDupgRfxJrTgKv): protocol: write tcp 188.166.225.170:57528->145.79.227.4:4189: use of closed network connection; write tcp 188.166.225.170:57528->145.79.227.4:4189: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57528->145.79.227.4:4189: use of closed network connection Dec 31 00:33:27.867723689 (609h4m27.738806724s) [noise=true , address=145.79.227.4:4189 , piece_id=ZZF4DSIPQBWKBD45UR2J5P75U5VVMTVB54NEX5QKZV4K2WJZ43BQ, seg_pos={0 0}, upload=533792]: Long tail cancellation due to optimal threshold Dec 31 00:33:27.867744699 (609h4m27.738827734s) [noise=true , address=145.79.227.2:4128 , piece_id=DW37Q32FFAZ37KHMOII6XGCJVDO2XTKED5TLOKG65JULWTXJIQEA, seg_pos={0 0}, upload=533792]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1wABkAtAjoYkKyNoAiaVCxSvY6WgvHvAYFPCjpNojtpiYiGtDS): protocol: expected piece hash; context canceled; EOF Dec 31 00:33:27.867753616 (609h4m27.738836651s) [noise=true , address=145.79.227.2:4128 , piece_id=DW37Q32FFAZ37KHMOII6XGCJVDO2XTKED5TLOKG65JULWTXJIQEA, seg_pos={0 0}, upload=533792]: Long tail cancellation due to optimal threshold Dec 31 00:33:27.867777462 (609h4m27.738860507s) [seg_pos={0 0}, upload=533792]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 31 00:33:27.867853687 (609h4m27.738936722s) [upload=533792]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 00:33:27.867859989 (609h4m27.738943024s) [upload=533792]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 00:33:28.167335511 (609h4m28.038418546s) [upload=533792]: Done uploading segments... Dec 31 00:33:28.167355899 (609h4m28.038438934s) [upload=533792]: Upload finished. err: <nil> Duration: 32.108752156s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 4.901677735s trace: https://jaeger.dev.storj.io/trace/63d344eb9b98ed45 FAIL (2025-12-31 00:39:56 UTC) orbiter warning: operation: DeleteObject uplink: metaclient: context deadline exceeded Duration: 4.901706861s core uplink eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 4.982426447s trace: https://jaeger.dev.storj.io/trace/63d344eb9b98ed45 FAIL (2025-12-31 00:39:01 UTC) orbiter warning: operation: DeleteObject uplink: metaclient: context deadline exceeded Duration: 4.98245917s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 27.131176954s trace: https://jaeger.dev.storj.io/trace/67f6164cdd15ab81 FAIL (2025-12-31 00:38:58 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== 31 00:33:25.026367244 (609h4m24.897450289s) [noise=true , address=145.79.227.2:4128 , piece_id=DW37Q32FFAZ37KHMOII6XGCJVDO2XTKED5TLOKG65JULWTXJIQEA, seg_pos={0 0}, upload=533792]: Uploading piece... Dec 31 00:33:25.026878247 (609h4m24.897961292s) [noise=true , address=145.79.227.44:4217 , piece_id=GW44B36EHWVCRLEM6BA5MXC3ITRNCCJATW2RXSYTNC7K7WRO2IDQ, seg_pos={0 0}, upload=533792]: Uploading piece... Dec 31 00:33:25.027303136 (609h4m24.898386171s) [noise=true , address=45.91.116.13:4161 , piece_id=K4CTNQGHL5S2GOARW2F42REGZ4QVWA2RKUNULONYX3VPVHAGONUQ, seg_pos={0 0}, upload=533792]: Uploading piece... Dec 31 00:33:25.027901605 (609h4m24.89898465s) [noise=true , address=77.90.31.13:4632 , piece_id=5KUEGWEJV5NJE2EX3S5S2G7TS5TK23ZQXV3WPA4LLHO65QJRVW6Q, seg_pos={0 0}, upload=533792]: Uploading piece... Dec 31 00:33:25.028257373 (609h4m24.899340408s) [noise=true , address=145.79.227.30:4769 , piece_id=P2ANRKDKCOTOMCZZMDIH2YRVI366UOHWO3LXV7INEDB2GW35Y52A, seg_pos={0 0}, upload=533792]: Uploading piece... Dec 31 00:33:25.028623710 (609h4m24.899706745s) [noise=true , address=145.79.227.34:4930 , piece_id=L3AVRBWPDFTZANZKO4TGITCA3WHBECKEQ2BGMLFQJYN4CR7FSADA, seg_pos={0 0}, upload=533792]: Uploading piece... Dec 31 00:33:25.028952256 (609h4m24.900035291s) [noise=true , address=145.79.227.25:4630 , piece_id=5BVJWWAW2QUQABWWZCD4EW54ZVH3K5RJNM5EXJJUGXI33O6CC24A, seg_pos={0 0}, upload=533792]: Uploading piece... Dec 31 00:33:25.029314766 (609h4m24.900397801s) [noise=true , address=145.79.227.29:4971 , piece_id=EIY6YRG5ZRMEW7IWE5X25RCTGNCU3HQ27O7Q434OAZ4EI335HU4A, seg_pos={0 0}, upload=533792]: Uploading piece... Dec 31 00:33:25.029659533 (609h4m24.900742578s) [noise=true , address=145.79.227.6:4283 , piece_id=MXSG7TVTQBR2AFONVXYWQB4DBAQEEXW226FDC6SSVG5TTWGPAWFQ, seg_pos={0 0}, upload=533792]: Uploading piece... Dec 31 00:33:25.030576118 (609h4m24.901659153s) [noise=true , address=77.90.31.18:4835 , piece_id=GRMJTN6ZDDYITHJWX5UGIAGLCOW6AKADHTK6UCVGRB437TUTDFGA, seg_pos={0 0}, upload=533792]: Uploading piece... Dec 31 00:33:25.031413452 (609h4m24.902496487s) [noise=true , address=64.49.13.155:4913 , piece_id=E4WJCMYJVURPHENWEWR46KOAUAOZU7FW7WLC53DAU44BE7R3CHMQ, seg_pos={0 0}, upload=533792]: Uploading piece... Dec 31 00:33:25.031987204 (609h4m24.903070239s) [noise=true , address=145.79.227.49:4370 , piece_id=52YCNJY7BLJ3N3HDWBZCM7PP2ZPHHBS4GV4QS4WM2AWSLYKB5BBA, seg_pos={0 0}, upload=533792]: Uploading piece... Dec 31 00:33:25.032849947 (609h4m24.903932993s) [noise=true , address=145.79.227.4:4189 , piece_id=ZZF4DSIPQBWKBD45UR2J5P75U5VVMTVB54NEX5QKZV4K2WJZ43BQ, seg_pos={0 0}, upload=533792]: Uploading piece... Dec 31 00:33:25.034222320 (609h4m24.905305355s) [noise=true , address=145.79.227.49:4353 , piece_id=MVTAW6QPJY2LRNEGISHNDP7AC5ZIIUR5N5BBZRFO5I6VYJU7UR3Q, seg_pos={0 0}, upload=533792]: Uploading piece... Dec 31 00:33:25.035359376 (609h4m24.906442411s) [noise=true , address=145.79.227.35:4494 , piece_id=EY2OJLQIF4CH7DQWJWXSBJMQI37NAXSBLQP4ZPMU3IBRVLWJVYWA, seg_pos={0 0}, upload=533792]: Uploading piece... Dec 31 00:33:25.035852344 (609h4m24.906935379s) [noise=true , address=145.79.227.33:4873 , piece_id=2DSEGSC5UWL76NEXSE7WXOGSQNRNXALAINY52TLQHB57LTS2R6ZQ, seg_pos={0 0}, upload=533792]: Uploading piece... Dec 31 00:33:26.491429069 (609h4m26.362512104s) [upload=533792]: Next returned nil segment Dec 31 00:33:26.491441904 (609h4m26.362524939s) [upload=533792]: Waiting for error group managing segments... Dec 31 00:33:26.876786140 (609h4m26.747869175s) [noise=true , address=145.79.227.6:4283 , piece_id=MXSG7TVTQBR2AFONVXYWQB4DBAQEEXW226FDC6SSVG5TTWGPAWFQ, seg_pos={0 0}, upload=533792]: Done uploading piece. err: <nil> Dec 31 00:33:26.895923271 (609h4m26.767006316s) [noise=true , address=145.79.227.35:4494 , piece_id=EY2OJLQIF4CH7DQWJWXSBJMQI37NAXSBLQP4ZPMU3IBRVLWJVYWA, seg_pos={0 0}, upload=533792]: Done uploading piece. err: <nil> Dec 31 00:33:26.903106888 (609h4m26.774189923s) [noise=true , address=145.79.227.33:4873 , piece_id=2DSEGSC5UWL76NEXSE7WXOGSQNRNXALAINY52TLQHB57LTS2R6ZQ, seg_pos={0 0}, upload=533792]: Done uploading piece. err: <nil> Dec 31 00:33:26.912484285 (609h4m26.78356733s) [noise=true , address=145.79.227.29:4971 , piece_id=EIY6YRG5ZRMEW7IWE5X25RCTGNCU3HQ27O7Q434OAZ4EI335HU4A, seg_pos={0 0}, upload=533792]: Done uploading piece. err: <nil> Dec 31 00:33:27.004280888 (609h4m26.875363933s) [noise=true , address=145.79.227.21:4495 , piece_id=YWETKWGBGTY3LALK4DXWSKUTM2TFEYB4N5A3KNKW346V3F5XRN2Q, seg_pos={0 0}, upload=533792]: Done uploading piece. err: <nil> Dec 31 00:33:27.118759862 (609h4m26.989842897s) [noise=true , address=77.90.31.13:4632 , piece_id=5KUEGWEJV5NJE2EX3S5S2G7TS5TK23ZQXV3WPA4LLHO65QJRVW6Q, seg_pos={0 0}, upload=533792]: Done uploading piece. err: <nil> Dec 31 00:33:27.225324635 (609h4m27.09640767s) [noise=true , address=45.91.116.13:4161 , piece_id=K4CTNQGHL5S2GOARW2F42REGZ4QVWA2RKUNULONYX3VPVHAGONUQ, seg_pos={0 0}, upload=533792]: Done uploading piece. err: <nil> Dec 31 00:33:27.347765488 (609h4m27.218848533s) [noise=true , address=45.91.116.12:4248 , piece_id=QXKIUKBDX2GSJ2EQPALFCLPPLNWBPL3GRAEHLPJNSWUDIMU7WHYA, seg_pos={0 0}, upload=533792]: Done uploading piece. err: <nil> Dec 31 00:33:27.789508217 (609h4m27.660591252s) [noise=true , address=145.79.227.44:4217 , piece_id=GW44B36EHWVCRLEM6BA5MXC3ITRNCCJATW2RXSYTNC7K7WRO2IDQ, seg_pos={0 0}, upload=533792]: Done uploading piece. err: <nil> Dec 31 00:33:27.821042021 (609h4m27.692125056s) [noise=true , address=145.79.227.49:4353 , piece_id=MVTAW6QPJY2LRNEGISHNDP7AC5ZIIUR5N5BBZRFO5I6VYJU7UR3Q, seg_pos={0 0}, upload=533792]: Done uploading piece. err: <nil> Dec 31 00:33:27.828648224 (609h4m27.699731259s) [noise=true , address=145.79.227.49:4370 , piece_id=52YCNJY7BLJ3N3HDWBZCM7PP2ZPHHBS4GV4QS4WM2AWSLYKB5BBA, seg_pos={0 0}, upload=533792]: Done uploading piece. err: <nil> Dec 31 00:33:27.840271497 (609h4m27.711354542s) [noise=true , address=145.79.227.30:4769 , piece_id=P2ANRKDKCOTOMCZZMDIH2YRVI366UOHWO3LXV7INEDB2GW35Y52A, seg_pos={0 0}, upload=533792]: Done uploading piece. err: <nil> Dec 31 00:33:27.844064569 (609h4m27.715147614s) [noise=true , address=145.79.227.34:4930 , piece_id=L3AVRBWPDFTZANZKO4TGITCA3WHBECKEQ2BGMLFQJYN4CR7FSADA, seg_pos={0 0}, upload=533792]: Done uploading piece. err: <nil> Dec 31 00:33:27.867248213 (609h4m27.738331248s) [noise=true , address=77.90.31.18:4835 , piece_id=GRMJTN6ZDDYITHJWX5UGIAGLCOW6AKADHTK6UCVGRB437TUTDFGA, seg_pos={0 0}, upload=533792]: Done uploading piece. err: <nil> Dec 31 00:33:27.867270586 (609h4m27.738353621s) [seg_pos={0 0}, upload=533792]: Segment reached optimal threshold of 14 pieces. Dec 31 00:33:27.867476859 (609h4m27.738559904s) [noise=true , address=64.49.13.155:4913 , piece_id=E4WJCMYJVURPHENWEWR46KOAUAOZU7FW7WLC53DAU44BE7R3CHMQ, seg_pos={0 0}, upload=533792]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oyYUnWDyFo1npaKpBxMG5K31oUzJnTpcU1XHFmKux3MCVUPZV): protocol: expected piece hash; context canceled; EOF Dec 31 00:33:27.867485555 (609h4m27.73856859s) [noise=true , address=64.49.13.155:4913 , piece_id=E4WJCMYJVURPHENWEWR46KOAUAOZU7FW7WLC53DAU44BE7R3CHMQ, seg_pos={0 0}, upload=533792]: Long tail cancellation due to optimal threshold Dec 31 00:33:27.867516204 (609h4m27.738599249s) [noise=true , address=145.79.227.25:4630 , piece_id=5BVJWWAW2QUQABWWZCD4EW54ZVH3K5RJNM5EXJJUGXI33O6CC24A, seg_pos={0 0}, upload=533792]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12aETKgTzKw8fyYqbeDA2JzfxZnWn12xUPYK1eRt6TvBPPdRJDp): protocol: expected piece hash; context canceled; EOF Dec 31 00:33:27.867526553 (609h4m27.738609588s) [noise=true , address=145.79.227.25:4630 , piece_id=5BVJWWAW2QUQABWWZCD4EW54ZVH3K5RJNM5EXJJUGXI33O6CC24A, seg_pos={0 0}, upload=533792]: Long tail cancellation due to optimal threshold Dec 31 00:33:27.867714432 (609h4m27.738797477s) [noise=true , address=145.79.227.4:4189 , piece_id=ZZF4DSIPQBWKBD45UR2J5P75U5VVMTVB54NEX5QKZV4K2WJZ43BQ, seg_pos={0 0}, upload=533792]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jH3757i2F3q18XSpoXznsysZPaXmEnqynpLPDupgRfxJrTgKv): protocol: write tcp 188.166.225.170:57528->145.79.227.4:4189: use of closed network connection; write tcp 188.166.225.170:57528->145.79.227.4:4189: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57528->145.79.227.4:4189: use of closed network connection Dec 31 00:33:27.867723689 (609h4m27.738806724s) [noise=true , address=145.79.227.4:4189 , piece_id=ZZF4DSIPQBWKBD45UR2J5P75U5VVMTVB54NEX5QKZV4K2WJZ43BQ, seg_pos={0 0}, upload=533792]: Long tail cancellation due to optimal threshold Dec 31 00:33:27.867744699 (609h4m27.738827734s) [noise=true , address=145.79.227.2:4128 , piece_id=DW37Q32FFAZ37KHMOII6XGCJVDO2XTKED5TLOKG65JULWTXJIQEA, seg_pos={0 0}, upload=533792]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1wABkAtAjoYkKyNoAiaVCxSvY6WgvHvAYFPCjpNojtpiYiGtDS): protocol: expected piece hash; context canceled; EOF Dec 31 00:33:27.867753616 (609h4m27.738836651s) [noise=true , address=145.79.227.2:4128 , piece_id=DW37Q32FFAZ37KHMOII6XGCJVDO2XTKED5TLOKG65JULWTXJIQEA, seg_pos={0 0}, upload=533792]: Long tail cancellation due to optimal threshold Dec 31 00:33:27.867777462 (609h4m27.738860507s) [seg_pos={0 0}, upload=533792]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 31 00:33:27.867853687 (609h4m27.738936722s) [upload=533792]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 00:33:27.867859989 (609h4m27.738943024s) [upload=533792]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 00:33:28.167335511 (609h4m28.038418546s) [upload=533792]: Done uploading segments... Dec 31 00:33:28.167355899 (609h4m28.038438934s) [upload=533792]: Upload finished. err: <nil> Duration: 27.131199897s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [..............X...X...........X........X..X....XX.XXX....X..] duration: 6.035286487s trace: https://jaeger.dev.storj.io/trace/4804197dbd063942 FAIL (2025-12-31 00:38:54 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, Piece ID: QRFMMAPSLAYNXMN5JG2PKJ5DZUGPOXUSDBYRBGCYR34WT63APUOA) context deadline exceeded; piecestore: (Node ID: 12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f, Piece ID: WAZYSDAN5MSE3C3UGJ5BQ7GDOPLPAOB7YQTHYUTIE7OZWRIVEARA) context deadline exceeded ======LOGS====== Dec 31 00:37:21.551769648 (609h8m21.422852683s) [upload=533848]: Starting upload Dec 31 00:37:21.551853758 (609h8m21.422936803s) [upload=533848]: Done starting upload Dec 31 00:37:21.551965832 (609h8m21.423048887s) [upload=533848]: Uploading segments... Dec 31 00:37:21.552177856 (609h8m21.423260901s) [upload=533848]: Got next segment. Inline: false Dec 31 00:37:21.552220857 (609h8m21.423303902s) [upload=533848]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 00:37:21.552226658 (609h8m21.423309703s) [upload=533848]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 00:37:21.804536710 (609h8m21.675619755s) [seg_pos={0 0}, upload=533848]: Begin upload segment... Dec 31 00:37:21.804586926 (609h8m21.675669961s) [seg_pos={0 0}, upload=533848]: Done begin upload segment. Dec 31 00:37:21.804611863 (609h8m21.675694908s) [noise=true , address=77.90.31.18:4805 , piece_id=QRFMMAPSLAYNXMN5JG2PKJ5DZUGPOXUSDBYRBGCYR34WT63APUOA, seg_pos={0 0}, upload=533848]: Uploading piece... Dec 31 00:37:21.804621622 (609h8m21.675704657s) [noise=true , address=77.90.31.21:4878 , piece_id=T6MNCWRE4CB45ARESCQV3UCXA6FHSBVSC6SFJCRA4FOKL4QJ7Y2A, seg_pos={0 0}, upload=533848]: Uploading piece... Dec 31 00:37:21.805095093 (609h8m21.676178138s) [noise=true , address=145.79.227.11:4732 , piece_id=NXZGC73AGULETYQ2EMVQ7XM36TNFNECSTUGTI3VMCPXJGGY2ALQA, seg_pos={0 0}, upload=533848]: Uploading piece... Dec 31 00:37:21.805232335 (609h8m21.67631537s) [noise=true , address=45.91.116.5:4299 , piece_id=2GPR5IUAX2AHYXUSSLGXOOU6C3PYEPIY6P4B67GBTMIE5X4FFVHQ, seg_pos={0 0}, upload=533848]: Uploading piece... Dec 31 00:37:21.805605015 (609h8m21.67668806s) [noise=true , address=145.79.227.41:4086 , piece_id=KCXGGAOKZNWFLLBV7I6P7LHVNM6DFRN5DITCURNMWBA2VE7CKVHA, seg_pos={0 0}, upload=533848]: Uploading piece... Dec 31 00:37:21.805935163 (609h8m21.677018198s) [noise=true , address=45.91.116.6:4975 , piece_id=AC5RI6DJ3L5QPO22FF3J3KQW2UEQVC5QBD247E35JIMIIQ5OQP4A, seg_pos={0 0}, upload=533848]: Uploading piece... Dec 31 00:37:21.805997943 (609h8m21.677080988s) [noise=true , address=45.91.116.8:4045 , piece_id=RTVLMBX2SLOK23HMAWEA4G6WHEEZEBH6XQRKKRK6WD5GD6BDW7NA, seg_pos={0 0}, upload=533848]: Uploading piece... Dec 31 00:37:21.806193325 (609h8m21.67727636s) [noise=true , address=145.79.227.31:4797 , piece_id=IZMS3LM7IG3RBXIMQTRDCINIQBQ4MNQM4XXDSJUTEJBBVTMSEZEA, seg_pos={0 0}, upload=533848]: Uploading piece... Dec 31 00:37:21.806390720 (609h8m21.677473765s) [noise=true , address=45.91.116.10:4081 , piece_id=6GJLWPKDJL6TUXLQLGS62F7YG6PPZ42O3VQT6KQFNTTBOZVSFDGQ, seg_pos={0 0}, upload=533848]: Uploading piece... Dec 31 00:37:21.806476133 (609h8m21.677559178s) [noise=true , address=145.79.227.43:4157 , piece_id=42TVBZYTZ4O3AJ5FHBZMPBMWMEUIZDGY5NXYHAZ3TBKNHG6QCNRQ, seg_pos={0 0}, upload=533848]: Uploading piece... Dec 31 00:37:21.806866076 (609h8m21.677949121s) [noise=true , address=145.79.227.19:4079 , piece_id=W2AGJA3O2XARCZEI5USBHZE6EIWIDAFU24WKPCUHQII65YQ2VAKA, seg_pos={0 0}, upload=533848]: Uploading piece... Dec 31 00:37:21.806964183 (609h8m21.678047228s) [noise=true , address=77.90.31.13:4629 , piece_id=WAZYSDAN5MSE3C3UGJ5BQ7GDOPLPAOB7YQTHYUTIE7OZWRIVEARA, seg_pos={0 0}, upload=533848]: Uploading piece... Dec 31 00:37:21.807173682 (609h8m21.678256717s) [noise=true , address=145.79.227.10:4402 , piece_id=UJXD3JBNBCKBK2H4OAW4KBCFXMZMP3MHDKYZOYFYSKCAJDWIU4YQ, seg_pos={0 0}, upload=533848]: Uploading piece... Dec 31 00:37:21.807465367 (609h8m21.678548402s) [noise=true , address=45.91.116.9:4262 , piece_id=6FNAPYOYWGGJYCTPXCMWI7L325YOIP6RDE7UZGJUH5XZICAA4CTQ, seg_pos={0 0}, upload=533848]: Uploading piece... Dec 31 00:37:21.807912939 (609h8m21.678995984s) [noise=true , address=77.90.31.14:4663 , piece_id=BEL6CMHHCPH6PJBFQBY2P22GI2DSHLU6C2FI3KREEKFXVESBJROQ, seg_pos={0 0}, upload=533848]: Uploading piece... Dec 31 00:37:21.808755323 (609h8m21.679838368s) [noise=true , address=145.79.227.29:4942 , piece_id=3M3YMDPBFYAW2RZGTFMZLPVGDP7HY72DLHR2PALDI6FJFYDLHFAA, seg_pos={0 0}, upload=533848]: Uploading piece... Dec 31 00:37:23.506877615 (609h8m23.37796065s) [upload=533848]: Next returned nil segment Dec 31 00:37:23.506893356 (609h8m23.377976401s) [upload=533848]: Waiting for error group managing segments... Dec 31 00:37:23.787196661 (609h8m23.658279696s) [noise=true , address=145.79.227.19:4079 , piece_id=W2AGJA3O2XARCZEI5USBHZE6EIWIDAFU24WKPCUHQII65YQ2VAKA, seg_pos={0 0}, upload=533848]: Done uploading piece. err: <nil> Dec 31 00:37:23.794707001 (609h8m23.665790046s) [noise=true , address=145.79.227.31:4797 , piece_id=IZMS3LM7IG3RBXIMQTRDCINIQBQ4MNQM4XXDSJUTEJBBVTMSEZEA, seg_pos={0 0}, upload=533848]: Done uploading piece. err: <nil> Dec 31 00:37:23.802772670 (609h8m23.673855705s) [noise=true , address=145.79.227.10:4402 , piece_id=UJXD3JBNBCKBK2H4OAW4KBCFXMZMP3MHDKYZOYFYSKCAJDWIU4YQ, seg_pos={0 0}, upload=533848]: Done uploading piece. err: <nil> Dec 31 00:37:23.809582226 (609h8m23.680665261s) [noise=true , address=145.79.227.11:4732 , piece_id=NXZGC73AGULETYQ2EMVQ7XM36TNFNECSTUGTI3VMCPXJGGY2ALQA, seg_pos={0 0}, upload=533848]: Done uploading piece. err: <nil> Dec 31 00:37:23.879984451 (609h8m23.751067486s) [noise=true , address=145.79.227.41:4086 , piece_id=KCXGGAOKZNWFLLBV7I6P7LHVNM6DFRN5DITCURNMWBA2VE7CKVHA, seg_pos={0 0}, upload=533848]: Done uploading piece. err: <nil> Dec 31 00:37:23.982919423 (609h8m23.854002458s) [noise=true , address=77.90.31.13:4629 , piece_id=WAZYSDAN5MSE3C3UGJ5BQ7GDOPLPAOB7YQTHYUTIE7OZWRIVEARA, seg_pos={0 0}, upload=533848]: Done uploading piece. err: <nil> Dec 31 00:37:24.025039846 (609h8m23.896122881s) [noise=true , address=77.90.31.18:4805 , piece_id=QRFMMAPSLAYNXMN5JG2PKJ5DZUGPOXUSDBYRBGCYR34WT63APUOA, seg_pos={0 0}, upload=533848]: Done uploading piece. err: <nil> Dec 31 00:37:24.081686509 (609h8m23.952769695s) [noise=true , address=45.91.116.8:4045 , piece_id=RTVLMBX2SLOK23HMAWEA4G6WHEEZEBH6XQRKKRK6WD5GD6BDW7NA, seg_pos={0 0}, upload=533848]: Done uploading piece. err: <nil> Dec 31 00:37:24.116560253 (609h8m23.987643288s) [noise=true , address=45.91.116.5:4299 , piece_id=2GPR5IUAX2AHYXUSSLGXOOU6C3PYEPIY6P4B67GBTMIE5X4FFVHQ, seg_pos={0 0}, upload=533848]: Done uploading piece. err: <nil> Dec 31 00:37:24.152081901 (609h8m24.023164936s) [noise=true , address=45.91.116.10:4081 , piece_id=6GJLWPKDJL6TUXLQLGS62F7YG6PPZ42O3VQT6KQFNTTBOZVSFDGQ, seg_pos={0 0}, upload=533848]: Done uploading piece. err: <nil> Dec 31 00:37:24.167484239 (609h8m24.038567284s) [noise=true , address=77.90.31.14:4663 , piece_id=BEL6CMHHCPH6PJBFQBY2P22GI2DSHLU6C2FI3KREEKFXVESBJROQ, seg_pos={0 0}, upload=533848]: Done uploading piece. err: <nil> Dec 31 00:37:24.199729909 (609h8m24.070812944s) [noise=true , address=45.91.116.9:4262 , piece_id=6FNAPYOYWGGJYCTPXCMWI7L325YOIP6RDE7UZGJUH5XZICAA4CTQ, seg_pos={0 0}, upload=533848]: Done uploading piece. err: <nil> Dec 31 00:37:24.728312881 (609h8m24.599395926s) [noise=true , address=77.90.31.21:4878 , piece_id=T6MNCWRE4CB45ARESCQV3UCXA6FHSBVSC6SFJCRA4FOKL4QJ7Y2A, seg_pos={0 0}, upload=533848]: Done uploading piece. err: <nil> Dec 31 00:37:24.728331858 (609h8m24.599414893s) [seg_pos={0 0}, upload=533848]: Segment reached optimal threshold of 13 pieces. Dec 31 00:37:24.728520977 (609h8m24.599604013s) [noise=true , address=145.79.227.43:4157 , piece_id=42TVBZYTZ4O3AJ5FHBZMPBMWMEUIZDGY5NXYHAZ3TBKNHG6QCNRQ, seg_pos={0 0}, upload=533848]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZrzADk2mKG4R9tjxUKjQeJvb4WmfjLksGFyvstHRfYbBC7XNz): protocol: expected piece hash; context canceled; EOF Dec 31 00:37:24.728527470 (609h8m24.599610505s) [noise=true , address=145.79.227.43:4157 , piece_id=42TVBZYTZ4O3AJ5FHBZMPBMWMEUIZDGY5NXYHAZ3TBKNHG6QCNRQ, seg_pos={0 0}, upload=533848]: Long tail cancellation due to optimal threshold Dec 31 00:37:24.728572045 (609h8m24.59965508s) [noise=true , address=145.79.227.29:4942 , piece_id=3M3YMDPBFYAW2RZGTFMZLPVGDP7HY72DLHR2PALDI6FJFYDLHFAA, seg_pos={0 0}, upload=533848]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17o881pUMuG7K2dv83k7RkrJCrmM7kgUiWesgam4GEw21KVUn7): protocol: expected piece hash; context canceled; EOF Dec 31 00:37:24.728580772 (609h8m24.599663807s) [noise=true , address=145.79.227.29:4942 , piece_id=3M3YMDPBFYAW2RZGTFMZLPVGDP7HY72DLHR2PALDI6FJFYDLHFAA, seg_pos={0 0}, upload=533848]: Long tail cancellation due to optimal threshold Dec 31 00:37:24.728718213 (609h8m24.599801258s) [noise=true , address=45.91.116.6:4975 , piece_id=AC5RI6DJ3L5QPO22FF3J3KQW2UEQVC5QBD247E35JIMIIQ5OQP4A, seg_pos={0 0}, upload=533848]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Dec 31 00:37:24.728724195 (609h8m24.59980723s) [noise=true , address=45.91.116.6:4975 , piece_id=AC5RI6DJ3L5QPO22FF3J3KQW2UEQVC5QBD247E35JIMIIQ5OQP4A, seg_pos={0 0}, upload=533848]: Long tail cancellation due to optimal threshold Dec 31 00:37:24.728742089 (609h8m24.599825124s) [seg_pos={0 0}, upload=533848]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 00:37:24.728824065 (609h8m24.5999071s) [upload=533848]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 00:37:24.728828854 (609h8m24.599911889s) [upload=533848]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 00:37:25.041591605 (609h8m24.91267466s) [upload=533848]: Done uploading segments... Dec 31 00:37:25.041615270 (609h8m24.912698315s) [upload=533848]: Upload finished. err: <nil> Duration: 6.035315684s core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 5.024850533s trace: https://jaeger.dev.storj.io/trace/63d344eb9b98ed45 FAIL (2025-12-31 00:38:36 UTC) orbiter warning: operation: DeleteObject uplink: metaclient: context deadline exceeded Duration: 5.024878356s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [...............X...X...........X........X..X....XX.XXX....X.] duration: 6.239382043s trace: https://jaeger.dev.storj.io/trace/4804197dbd063942 FAIL (2025-12-31 00:38:32 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, Piece ID: QRFMMAPSLAYNXMN5JG2PKJ5DZUGPOXUSDBYRBGCYR34WT63APUOA) context deadline exceeded; piecestore: (Node ID: 12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f, Piece ID: WAZYSDAN5MSE3C3UGJ5BQ7GDOPLPAOB7YQTHYUTIE7OZWRIVEARA) context deadline exceeded ======LOGS====== Dec 31 00:37:21.551769648 (609h8m21.422852683s) [upload=533848]: Starting upload Dec 31 00:37:21.551853758 (609h8m21.422936803s) [upload=533848]: Done starting upload Dec 31 00:37:21.551965832 (609h8m21.423048887s) [upload=533848]: Uploading segments... Dec 31 00:37:21.552177856 (609h8m21.423260901s) [upload=533848]: Got next segment. Inline: false Dec 31 00:37:21.552220857 (609h8m21.423303902s) [upload=533848]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 00:37:21.552226658 (609h8m21.423309703s) [upload=533848]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 00:37:21.804536710 (609h8m21.675619755s) [seg_pos={0 0}, upload=533848]: Begin upload segment... Dec 31 00:37:21.804586926 (609h8m21.675669961s) [seg_pos={0 0}, upload=533848]: Done begin upload segment. Dec 31 00:37:21.804611863 (609h8m21.675694908s) [noise=true , address=77.90.31.18:4805 , piece_id=QRFMMAPSLAYNXMN5JG2PKJ5DZUGPOXUSDBYRBGCYR34WT63APUOA, seg_pos={0 0}, upload=533848]: Uploading piece... Dec 31 00:37:21.804621622 (609h8m21.675704657s) [noise=true , address=77.90.31.21:4878 , piece_id=T6MNCWRE4CB45ARESCQV3UCXA6FHSBVSC6SFJCRA4FOKL4QJ7Y2A, seg_pos={0 0}, upload=533848]: Uploading piece... Dec 31 00:37:21.805095093 (609h8m21.676178138s) [noise=true , address=145.79.227.11:4732 , piece_id=NXZGC73AGULETYQ2EMVQ7XM36TNFNECSTUGTI3VMCPXJGGY2ALQA, seg_pos={0 0}, upload=533848]: Uploading piece... Dec 31 00:37:21.805232335 (609h8m21.67631537s) [noise=true , address=45.91.116.5:4299 , piece_id=2GPR5IUAX2AHYXUSSLGXOOU6C3PYEPIY6P4B67GBTMIE5X4FFVHQ, seg_pos={0 0}, upload=533848]: Uploading piece... Dec 31 00:37:21.805605015 (609h8m21.67668806s) [noise=true , address=145.79.227.41:4086 , piece_id=KCXGGAOKZNWFLLBV7I6P7LHVNM6DFRN5DITCURNMWBA2VE7CKVHA, seg_pos={0 0}, upload=533848]: Uploading piece... Dec 31 00:37:21.805935163 (609h8m21.677018198s) [noise=true , address=45.91.116.6:4975 , piece_id=AC5RI6DJ3L5QPO22FF3J3KQW2UEQVC5QBD247E35JIMIIQ5OQP4A, seg_pos={0 0}, upload=533848]: Uploading piece... Dec 31 00:37:21.805997943 (609h8m21.677080988s) [noise=true , address=45.91.116.8:4045 , piece_id=RTVLMBX2SLOK23HMAWEA4G6WHEEZEBH6XQRKKRK6WD5GD6BDW7NA, seg_pos={0 0}, upload=533848]: Uploading piece... Dec 31 00:37:21.806193325 (609h8m21.67727636s) [noise=true , address=145.79.227.31:4797 , piece_id=IZMS3LM7IG3RBXIMQTRDCINIQBQ4MNQM4XXDSJUTEJBBVTMSEZEA, seg_pos={0 0}, upload=533848]: Uploading piece... Dec 31 00:37:21.806390720 (609h8m21.677473765s) [noise=true , address=45.91.116.10:4081 , piece_id=6GJLWPKDJL6TUXLQLGS62F7YG6PPZ42O3VQT6KQFNTTBOZVSFDGQ, seg_pos={0 0}, upload=533848]: Uploading piece... Dec 31 00:37:21.806476133 (609h8m21.677559178s) [noise=true , address=145.79.227.43:4157 , piece_id=42TVBZYTZ4O3AJ5FHBZMPBMWMEUIZDGY5NXYHAZ3TBKNHG6QCNRQ, seg_pos={0 0}, upload=533848]: Uploading piece... Dec 31 00:37:21.806866076 (609h8m21.677949121s) [noise=true , address=145.79.227.19:4079 , piece_id=W2AGJA3O2XARCZEI5USBHZE6EIWIDAFU24WKPCUHQII65YQ2VAKA, seg_pos={0 0}, upload=533848]: Uploading piece... Dec 31 00:37:21.806964183 (609h8m21.678047228s) [noise=true , address=77.90.31.13:4629 , piece_id=WAZYSDAN5MSE3C3UGJ5BQ7GDOPLPAOB7YQTHYUTIE7OZWRIVEARA, seg_pos={0 0}, upload=533848]: Uploading piece... Dec 31 00:37:21.807173682 (609h8m21.678256717s) [noise=true , address=145.79.227.10:4402 , piece_id=UJXD3JBNBCKBK2H4OAW4KBCFXMZMP3MHDKYZOYFYSKCAJDWIU4YQ, seg_pos={0 0}, upload=533848]: Uploading piece... Dec 31 00:37:21.807465367 (609h8m21.678548402s) [noise=true , address=45.91.116.9:4262 , piece_id=6FNAPYOYWGGJYCTPXCMWI7L325YOIP6RDE7UZGJUH5XZICAA4CTQ, seg_pos={0 0}, upload=533848]: Uploading piece... Dec 31 00:37:21.807912939 (609h8m21.678995984s) [noise=true , address=77.90.31.14:4663 , piece_id=BEL6CMHHCPH6PJBFQBY2P22GI2DSHLU6C2FI3KREEKFXVESBJROQ, seg_pos={0 0}, upload=533848]: Uploading piece... Dec 31 00:37:21.808755323 (609h8m21.679838368s) [noise=true , address=145.79.227.29:4942 , piece_id=3M3YMDPBFYAW2RZGTFMZLPVGDP7HY72DLHR2PALDI6FJFYDLHFAA, seg_pos={0 0}, upload=533848]: Uploading piece... Dec 31 00:37:23.506877615 (609h8m23.37796065s) [upload=533848]: Next returned nil segment Dec 31 00:37:23.506893356 (609h8m23.377976401s) [upload=533848]: Waiting for error group managing segments... Dec 31 00:37:23.787196661 (609h8m23.658279696s) [noise=true , address=145.79.227.19:4079 , piece_id=W2AGJA3O2XARCZEI5USBHZE6EIWIDAFU24WKPCUHQII65YQ2VAKA, seg_pos={0 0}, upload=533848]: Done uploading piece. err: <nil> Dec 31 00:37:23.794707001 (609h8m23.665790046s) [noise=true , address=145.79.227.31:4797 , piece_id=IZMS3LM7IG3RBXIMQTRDCINIQBQ4MNQM4XXDSJUTEJBBVTMSEZEA, seg_pos={0 0}, upload=533848]: Done uploading piece. err: <nil> Dec 31 00:37:23.802772670 (609h8m23.673855705s) [noise=true , address=145.79.227.10:4402 , piece_id=UJXD3JBNBCKBK2H4OAW4KBCFXMZMP3MHDKYZOYFYSKCAJDWIU4YQ, seg_pos={0 0}, upload=533848]: Done uploading piece. err: <nil> Dec 31 00:37:23.809582226 (609h8m23.680665261s) [noise=true , address=145.79.227.11:4732 , piece_id=NXZGC73AGULETYQ2EMVQ7XM36TNFNECSTUGTI3VMCPXJGGY2ALQA, seg_pos={0 0}, upload=533848]: Done uploading piece. err: <nil> Dec 31 00:37:23.879984451 (609h8m23.751067486s) [noise=true , address=145.79.227.41:4086 , piece_id=KCXGGAOKZNWFLLBV7I6P7LHVNM6DFRN5DITCURNMWBA2VE7CKVHA, seg_pos={0 0}, upload=533848]: Done uploading piece. err: <nil> Dec 31 00:37:23.982919423 (609h8m23.854002458s) [noise=true , address=77.90.31.13:4629 , piece_id=WAZYSDAN5MSE3C3UGJ5BQ7GDOPLPAOB7YQTHYUTIE7OZWRIVEARA, seg_pos={0 0}, upload=533848]: Done uploading piece. err: <nil> Dec 31 00:37:24.025039846 (609h8m23.896122881s) [noise=true , address=77.90.31.18:4805 , piece_id=QRFMMAPSLAYNXMN5JG2PKJ5DZUGPOXUSDBYRBGCYR34WT63APUOA, seg_pos={0 0}, upload=533848]: Done uploading piece. err: <nil> Dec 31 00:37:24.081686509 (609h8m23.952769695s) [noise=true , address=45.91.116.8:4045 , piece_id=RTVLMBX2SLOK23HMAWEA4G6WHEEZEBH6XQRKKRK6WD5GD6BDW7NA, seg_pos={0 0}, upload=533848]: Done uploading piece. err: <nil> Dec 31 00:37:24.116560253 (609h8m23.987643288s) [noise=true , address=45.91.116.5:4299 , piece_id=2GPR5IUAX2AHYXUSSLGXOOU6C3PYEPIY6P4B67GBTMIE5X4FFVHQ, seg_pos={0 0}, upload=533848]: Done uploading piece. err: <nil> Dec 31 00:37:24.152081901 (609h8m24.023164936s) [noise=true , address=45.91.116.10:4081 , piece_id=6GJLWPKDJL6TUXLQLGS62F7YG6PPZ42O3VQT6KQFNTTBOZVSFDGQ, seg_pos={0 0}, upload=533848]: Done uploading piece. err: <nil> Dec 31 00:37:24.167484239 (609h8m24.038567284s) [noise=true , address=77.90.31.14:4663 , piece_id=BEL6CMHHCPH6PJBFQBY2P22GI2DSHLU6C2FI3KREEKFXVESBJROQ, seg_pos={0 0}, upload=533848]: Done uploading piece. err: <nil> Dec 31 00:37:24.199729909 (609h8m24.070812944s) [noise=true , address=45.91.116.9:4262 , piece_id=6FNAPYOYWGGJYCTPXCMWI7L325YOIP6RDE7UZGJUH5XZICAA4CTQ, seg_pos={0 0}, upload=533848]: Done uploading piece. err: <nil> Dec 31 00:37:24.728312881 (609h8m24.599395926s) [noise=true , address=77.90.31.21:4878 , piece_id=T6MNCWRE4CB45ARESCQV3UCXA6FHSBVSC6SFJCRA4FOKL4QJ7Y2A, seg_pos={0 0}, upload=533848]: Done uploading piece. err: <nil> Dec 31 00:37:24.728331858 (609h8m24.599414893s) [seg_pos={0 0}, upload=533848]: Segment reached optimal threshold of 13 pieces. Dec 31 00:37:24.728520977 (609h8m24.599604013s) [noise=true , address=145.79.227.43:4157 , piece_id=42TVBZYTZ4O3AJ5FHBZMPBMWMEUIZDGY5NXYHAZ3TBKNHG6QCNRQ, seg_pos={0 0}, upload=533848]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZrzADk2mKG4R9tjxUKjQeJvb4WmfjLksGFyvstHRfYbBC7XNz): protocol: expected piece hash; context canceled; EOF Dec 31 00:37:24.728527470 (609h8m24.599610505s) [noise=true , address=145.79.227.43:4157 , piece_id=42TVBZYTZ4O3AJ5FHBZMPBMWMEUIZDGY5NXYHAZ3TBKNHG6QCNRQ, seg_pos={0 0}, upload=533848]: Long tail cancellation due to optimal threshold Dec 31 00:37:24.728572045 (609h8m24.59965508s) [noise=true , address=145.79.227.29:4942 , piece_id=3M3YMDPBFYAW2RZGTFMZLPVGDP7HY72DLHR2PALDI6FJFYDLHFAA, seg_pos={0 0}, upload=533848]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17o881pUMuG7K2dv83k7RkrJCrmM7kgUiWesgam4GEw21KVUn7): protocol: expected piece hash; context canceled; EOF Dec 31 00:37:24.728580772 (609h8m24.599663807s) [noise=true , address=145.79.227.29:4942 , piece_id=3M3YMDPBFYAW2RZGTFMZLPVGDP7HY72DLHR2PALDI6FJFYDLHFAA, seg_pos={0 0}, upload=533848]: Long tail cancellation due to optimal threshold Dec 31 00:37:24.728718213 (609h8m24.599801258s) [noise=true , address=45.91.116.6:4975 , piece_id=AC5RI6DJ3L5QPO22FF3J3KQW2UEQVC5QBD247E35JIMIIQ5OQP4A, seg_pos={0 0}, upload=533848]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Dec 31 00:37:24.728724195 (609h8m24.59980723s) [noise=true , address=45.91.116.6:4975 , piece_id=AC5RI6DJ3L5QPO22FF3J3KQW2UEQVC5QBD247E35JIMIIQ5OQP4A, seg_pos={0 0}, upload=533848]: Long tail cancellation due to optimal threshold Dec 31 00:37:24.728742089 (609h8m24.599825124s) [seg_pos={0 0}, upload=533848]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 00:37:24.728824065 (609h8m24.5999071s) [upload=533848]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 00:37:24.728828854 (609h8m24.599911889s) [upload=533848]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 00:37:25.041591605 (609h8m24.91267466s) [upload=533848]: Done uploading segments... Dec 31 00:37:25.041615270 (609h8m24.912698315s) [upload=533848]: Upload finished. err: <nil> Duration: 6.239408433s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 47.342181098s trace: https://jaeger.dev.storj.io/trace/67f6164cdd15ab81 FAIL (2025-12-31 00:38:24 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== 31 00:33:25.026367244 (609h4m24.897450289s) [noise=true , address=145.79.227.2:4128 , piece_id=DW37Q32FFAZ37KHMOII6XGCJVDO2XTKED5TLOKG65JULWTXJIQEA, seg_pos={0 0}, upload=533792]: Uploading piece... Dec 31 00:33:25.026878247 (609h4m24.897961292s) [noise=true , address=145.79.227.44:4217 , piece_id=GW44B36EHWVCRLEM6BA5MXC3ITRNCCJATW2RXSYTNC7K7WRO2IDQ, seg_pos={0 0}, upload=533792]: Uploading piece... Dec 31 00:33:25.027303136 (609h4m24.898386171s) [noise=true , address=45.91.116.13:4161 , piece_id=K4CTNQGHL5S2GOARW2F42REGZ4QVWA2RKUNULONYX3VPVHAGONUQ, seg_pos={0 0}, upload=533792]: Uploading piece... Dec 31 00:33:25.027901605 (609h4m24.89898465s) [noise=true , address=77.90.31.13:4632 , piece_id=5KUEGWEJV5NJE2EX3S5S2G7TS5TK23ZQXV3WPA4LLHO65QJRVW6Q, seg_pos={0 0}, upload=533792]: Uploading piece... Dec 31 00:33:25.028257373 (609h4m24.899340408s) [noise=true , address=145.79.227.30:4769 , piece_id=P2ANRKDKCOTOMCZZMDIH2YRVI366UOHWO3LXV7INEDB2GW35Y52A, seg_pos={0 0}, upload=533792]: Uploading piece... Dec 31 00:33:25.028623710 (609h4m24.899706745s) [noise=true , address=145.79.227.34:4930 , piece_id=L3AVRBWPDFTZANZKO4TGITCA3WHBECKEQ2BGMLFQJYN4CR7FSADA, seg_pos={0 0}, upload=533792]: Uploading piece... Dec 31 00:33:25.028952256 (609h4m24.900035291s) [noise=true , address=145.79.227.25:4630 , piece_id=5BVJWWAW2QUQABWWZCD4EW54ZVH3K5RJNM5EXJJUGXI33O6CC24A, seg_pos={0 0}, upload=533792]: Uploading piece... Dec 31 00:33:25.029314766 (609h4m24.900397801s) [noise=true , address=145.79.227.29:4971 , piece_id=EIY6YRG5ZRMEW7IWE5X25RCTGNCU3HQ27O7Q434OAZ4EI335HU4A, seg_pos={0 0}, upload=533792]: Uploading piece... Dec 31 00:33:25.029659533 (609h4m24.900742578s) [noise=true , address=145.79.227.6:4283 , piece_id=MXSG7TVTQBR2AFONVXYWQB4DBAQEEXW226FDC6SSVG5TTWGPAWFQ, seg_pos={0 0}, upload=533792]: Uploading piece... Dec 31 00:33:25.030576118 (609h4m24.901659153s) [noise=true , address=77.90.31.18:4835 , piece_id=GRMJTN6ZDDYITHJWX5UGIAGLCOW6AKADHTK6UCVGRB437TUTDFGA, seg_pos={0 0}, upload=533792]: Uploading piece... Dec 31 00:33:25.031413452 (609h4m24.902496487s) [noise=true , address=64.49.13.155:4913 , piece_id=E4WJCMYJVURPHENWEWR46KOAUAOZU7FW7WLC53DAU44BE7R3CHMQ, seg_pos={0 0}, upload=533792]: Uploading piece... Dec 31 00:33:25.031987204 (609h4m24.903070239s) [noise=true , address=145.79.227.49:4370 , piece_id=52YCNJY7BLJ3N3HDWBZCM7PP2ZPHHBS4GV4QS4WM2AWSLYKB5BBA, seg_pos={0 0}, upload=533792]: Uploading piece... Dec 31 00:33:25.032849947 (609h4m24.903932993s) [noise=true , address=145.79.227.4:4189 , piece_id=ZZF4DSIPQBWKBD45UR2J5P75U5VVMTVB54NEX5QKZV4K2WJZ43BQ, seg_pos={0 0}, upload=533792]: Uploading piece... Dec 31 00:33:25.034222320 (609h4m24.905305355s) [noise=true , address=145.79.227.49:4353 , piece_id=MVTAW6QPJY2LRNEGISHNDP7AC5ZIIUR5N5BBZRFO5I6VYJU7UR3Q, seg_pos={0 0}, upload=533792]: Uploading piece... Dec 31 00:33:25.035359376 (609h4m24.906442411s) [noise=true , address=145.79.227.35:4494 , piece_id=EY2OJLQIF4CH7DQWJWXSBJMQI37NAXSBLQP4ZPMU3IBRVLWJVYWA, seg_pos={0 0}, upload=533792]: Uploading piece... Dec 31 00:33:25.035852344 (609h4m24.906935379s) [noise=true , address=145.79.227.33:4873 , piece_id=2DSEGSC5UWL76NEXSE7WXOGSQNRNXALAINY52TLQHB57LTS2R6ZQ, seg_pos={0 0}, upload=533792]: Uploading piece... Dec 31 00:33:26.491429069 (609h4m26.362512104s) [upload=533792]: Next returned nil segment Dec 31 00:33:26.491441904 (609h4m26.362524939s) [upload=533792]: Waiting for error group managing segments... Dec 31 00:33:26.876786140 (609h4m26.747869175s) [noise=true , address=145.79.227.6:4283 , piece_id=MXSG7TVTQBR2AFONVXYWQB4DBAQEEXW226FDC6SSVG5TTWGPAWFQ, seg_pos={0 0}, upload=533792]: Done uploading piece. err: <nil> Dec 31 00:33:26.895923271 (609h4m26.767006316s) [noise=true , address=145.79.227.35:4494 , piece_id=EY2OJLQIF4CH7DQWJWXSBJMQI37NAXSBLQP4ZPMU3IBRVLWJVYWA, seg_pos={0 0}, upload=533792]: Done uploading piece. err: <nil> Dec 31 00:33:26.903106888 (609h4m26.774189923s) [noise=true , address=145.79.227.33:4873 , piece_id=2DSEGSC5UWL76NEXSE7WXOGSQNRNXALAINY52TLQHB57LTS2R6ZQ, seg_pos={0 0}, upload=533792]: Done uploading piece. err: <nil> Dec 31 00:33:26.912484285 (609h4m26.78356733s) [noise=true , address=145.79.227.29:4971 , piece_id=EIY6YRG5ZRMEW7IWE5X25RCTGNCU3HQ27O7Q434OAZ4EI335HU4A, seg_pos={0 0}, upload=533792]: Done uploading piece. err: <nil> Dec 31 00:33:27.004280888 (609h4m26.875363933s) [noise=true , address=145.79.227.21:4495 , piece_id=YWETKWGBGTY3LALK4DXWSKUTM2TFEYB4N5A3KNKW346V3F5XRN2Q, seg_pos={0 0}, upload=533792]: Done uploading piece. err: <nil> Dec 31 00:33:27.118759862 (609h4m26.989842897s) [noise=true , address=77.90.31.13:4632 , piece_id=5KUEGWEJV5NJE2EX3S5S2G7TS5TK23ZQXV3WPA4LLHO65QJRVW6Q, seg_pos={0 0}, upload=533792]: Done uploading piece. err: <nil> Dec 31 00:33:27.225324635 (609h4m27.09640767s) [noise=true , address=45.91.116.13:4161 , piece_id=K4CTNQGHL5S2GOARW2F42REGZ4QVWA2RKUNULONYX3VPVHAGONUQ, seg_pos={0 0}, upload=533792]: Done uploading piece. err: <nil> Dec 31 00:33:27.347765488 (609h4m27.218848533s) [noise=true , address=45.91.116.12:4248 , piece_id=QXKIUKBDX2GSJ2EQPALFCLPPLNWBPL3GRAEHLPJNSWUDIMU7WHYA, seg_pos={0 0}, upload=533792]: Done uploading piece. err: <nil> Dec 31 00:33:27.789508217 (609h4m27.660591252s) [noise=true , address=145.79.227.44:4217 , piece_id=GW44B36EHWVCRLEM6BA5MXC3ITRNCCJATW2RXSYTNC7K7WRO2IDQ, seg_pos={0 0}, upload=533792]: Done uploading piece. err: <nil> Dec 31 00:33:27.821042021 (609h4m27.692125056s) [noise=true , address=145.79.227.49:4353 , piece_id=MVTAW6QPJY2LRNEGISHNDP7AC5ZIIUR5N5BBZRFO5I6VYJU7UR3Q, seg_pos={0 0}, upload=533792]: Done uploading piece. err: <nil> Dec 31 00:33:27.828648224 (609h4m27.699731259s) [noise=true , address=145.79.227.49:4370 , piece_id=52YCNJY7BLJ3N3HDWBZCM7PP2ZPHHBS4GV4QS4WM2AWSLYKB5BBA, seg_pos={0 0}, upload=533792]: Done uploading piece. err: <nil> Dec 31 00:33:27.840271497 (609h4m27.711354542s) [noise=true , address=145.79.227.30:4769 , piece_id=P2ANRKDKCOTOMCZZMDIH2YRVI366UOHWO3LXV7INEDB2GW35Y52A, seg_pos={0 0}, upload=533792]: Done uploading piece. err: <nil> Dec 31 00:33:27.844064569 (609h4m27.715147614s) [noise=true , address=145.79.227.34:4930 , piece_id=L3AVRBWPDFTZANZKO4TGITCA3WHBECKEQ2BGMLFQJYN4CR7FSADA, seg_pos={0 0}, upload=533792]: Done uploading piece. err: <nil> Dec 31 00:33:27.867248213 (609h4m27.738331248s) [noise=true , address=77.90.31.18:4835 , piece_id=GRMJTN6ZDDYITHJWX5UGIAGLCOW6AKADHTK6UCVGRB437TUTDFGA, seg_pos={0 0}, upload=533792]: Done uploading piece. err: <nil> Dec 31 00:33:27.867270586 (609h4m27.738353621s) [seg_pos={0 0}, upload=533792]: Segment reached optimal threshold of 14 pieces. Dec 31 00:33:27.867476859 (609h4m27.738559904s) [noise=true , address=64.49.13.155:4913 , piece_id=E4WJCMYJVURPHENWEWR46KOAUAOZU7FW7WLC53DAU44BE7R3CHMQ, seg_pos={0 0}, upload=533792]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oyYUnWDyFo1npaKpBxMG5K31oUzJnTpcU1XHFmKux3MCVUPZV): protocol: expected piece hash; context canceled; EOF Dec 31 00:33:27.867485555 (609h4m27.73856859s) [noise=true , address=64.49.13.155:4913 , piece_id=E4WJCMYJVURPHENWEWR46KOAUAOZU7FW7WLC53DAU44BE7R3CHMQ, seg_pos={0 0}, upload=533792]: Long tail cancellation due to optimal threshold Dec 31 00:33:27.867516204 (609h4m27.738599249s) [noise=true , address=145.79.227.25:4630 , piece_id=5BVJWWAW2QUQABWWZCD4EW54ZVH3K5RJNM5EXJJUGXI33O6CC24A, seg_pos={0 0}, upload=533792]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12aETKgTzKw8fyYqbeDA2JzfxZnWn12xUPYK1eRt6TvBPPdRJDp): protocol: expected piece hash; context canceled; EOF Dec 31 00:33:27.867526553 (609h4m27.738609588s) [noise=true , address=145.79.227.25:4630 , piece_id=5BVJWWAW2QUQABWWZCD4EW54ZVH3K5RJNM5EXJJUGXI33O6CC24A, seg_pos={0 0}, upload=533792]: Long tail cancellation due to optimal threshold Dec 31 00:33:27.867714432 (609h4m27.738797477s) [noise=true , address=145.79.227.4:4189 , piece_id=ZZF4DSIPQBWKBD45UR2J5P75U5VVMTVB54NEX5QKZV4K2WJZ43BQ, seg_pos={0 0}, upload=533792]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jH3757i2F3q18XSpoXznsysZPaXmEnqynpLPDupgRfxJrTgKv): protocol: write tcp 188.166.225.170:57528->145.79.227.4:4189: use of closed network connection; write tcp 188.166.225.170:57528->145.79.227.4:4189: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57528->145.79.227.4:4189: use of closed network connection Dec 31 00:33:27.867723689 (609h4m27.738806724s) [noise=true , address=145.79.227.4:4189 , piece_id=ZZF4DSIPQBWKBD45UR2J5P75U5VVMTVB54NEX5QKZV4K2WJZ43BQ, seg_pos={0 0}, upload=533792]: Long tail cancellation due to optimal threshold Dec 31 00:33:27.867744699 (609h4m27.738827734s) [noise=true , address=145.79.227.2:4128 , piece_id=DW37Q32FFAZ37KHMOII6XGCJVDO2XTKED5TLOKG65JULWTXJIQEA, seg_pos={0 0}, upload=533792]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1wABkAtAjoYkKyNoAiaVCxSvY6WgvHvAYFPCjpNojtpiYiGtDS): protocol: expected piece hash; context canceled; EOF Dec 31 00:33:27.867753616 (609h4m27.738836651s) [noise=true , address=145.79.227.2:4128 , piece_id=DW37Q32FFAZ37KHMOII6XGCJVDO2XTKED5TLOKG65JULWTXJIQEA, seg_pos={0 0}, upload=533792]: Long tail cancellation due to optimal threshold Dec 31 00:33:27.867777462 (609h4m27.738860507s) [seg_pos={0 0}, upload=533792]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 31 00:33:27.867853687 (609h4m27.738936722s) [upload=533792]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 00:33:27.867859989 (609h4m27.738943024s) [upload=533792]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 00:33:28.167335511 (609h4m28.038418546s) [upload=533792]: Done uploading segments... Dec 31 00:33:28.167355899 (609h4m28.038438934s) [upload=533792]: Upload finished. err: <nil> Duration: 47.342208571s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [................X...X...........X........X..X....XX.XXX....X] duration: 30.002547091s trace: https://jaeger.dev.storj.io/trace/4804197dbd063942 FAIL (2025-12-31 00:37:50 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, Piece ID: QRFMMAPSLAYNXMN5JG2PKJ5DZUGPOXUSDBYRBGCYR34WT63APUOA) context deadline exceeded; piecestore: (Node ID: 12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f, Piece ID: WAZYSDAN5MSE3C3UGJ5BQ7GDOPLPAOB7YQTHYUTIE7OZWRIVEARA) context deadline exceeded ======LOGS====== Dec 31 00:37:21.551769648 (609h8m21.422852683s) [upload=533848]: Starting upload Dec 31 00:37:21.551853758 (609h8m21.422936803s) [upload=533848]: Done starting upload Dec 31 00:37:21.551965832 (609h8m21.423048887s) [upload=533848]: Uploading segments... Dec 31 00:37:21.552177856 (609h8m21.423260901s) [upload=533848]: Got next segment. Inline: false Dec 31 00:37:21.552220857 (609h8m21.423303902s) [upload=533848]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 00:37:21.552226658 (609h8m21.423309703s) [upload=533848]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 00:37:21.804536710 (609h8m21.675619755s) [seg_pos={0 0}, upload=533848]: Begin upload segment... Dec 31 00:37:21.804586926 (609h8m21.675669961s) [seg_pos={0 0}, upload=533848]: Done begin upload segment. Dec 31 00:37:21.804611863 (609h8m21.675694908s) [noise=true , address=77.90.31.18:4805 , piece_id=QRFMMAPSLAYNXMN5JG2PKJ5DZUGPOXUSDBYRBGCYR34WT63APUOA, seg_pos={0 0}, upload=533848]: Uploading piece... Dec 31 00:37:21.804621622 (609h8m21.675704657s) [noise=true , address=77.90.31.21:4878 , piece_id=T6MNCWRE4CB45ARESCQV3UCXA6FHSBVSC6SFJCRA4FOKL4QJ7Y2A, seg_pos={0 0}, upload=533848]: Uploading piece... Dec 31 00:37:21.805095093 (609h8m21.676178138s) [noise=true , address=145.79.227.11:4732 , piece_id=NXZGC73AGULETYQ2EMVQ7XM36TNFNECSTUGTI3VMCPXJGGY2ALQA, seg_pos={0 0}, upload=533848]: Uploading piece... Dec 31 00:37:21.805232335 (609h8m21.67631537s) [noise=true , address=45.91.116.5:4299 , piece_id=2GPR5IUAX2AHYXUSSLGXOOU6C3PYEPIY6P4B67GBTMIE5X4FFVHQ, seg_pos={0 0}, upload=533848]: Uploading piece... Dec 31 00:37:21.805605015 (609h8m21.67668806s) [noise=true , address=145.79.227.41:4086 , piece_id=KCXGGAOKZNWFLLBV7I6P7LHVNM6DFRN5DITCURNMWBA2VE7CKVHA, seg_pos={0 0}, upload=533848]: Uploading piece... Dec 31 00:37:21.805935163 (609h8m21.677018198s) [noise=true , address=45.91.116.6:4975 , piece_id=AC5RI6DJ3L5QPO22FF3J3KQW2UEQVC5QBD247E35JIMIIQ5OQP4A, seg_pos={0 0}, upload=533848]: Uploading piece... Dec 31 00:37:21.805997943 (609h8m21.677080988s) [noise=true , address=45.91.116.8:4045 , piece_id=RTVLMBX2SLOK23HMAWEA4G6WHEEZEBH6XQRKKRK6WD5GD6BDW7NA, seg_pos={0 0}, upload=533848]: Uploading piece... Dec 31 00:37:21.806193325 (609h8m21.67727636s) [noise=true , address=145.79.227.31:4797 , piece_id=IZMS3LM7IG3RBXIMQTRDCINIQBQ4MNQM4XXDSJUTEJBBVTMSEZEA, seg_pos={0 0}, upload=533848]: Uploading piece... Dec 31 00:37:21.806390720 (609h8m21.677473765s) [noise=true , address=45.91.116.10:4081 , piece_id=6GJLWPKDJL6TUXLQLGS62F7YG6PPZ42O3VQT6KQFNTTBOZVSFDGQ, seg_pos={0 0}, upload=533848]: Uploading piece... Dec 31 00:37:21.806476133 (609h8m21.677559178s) [noise=true , address=145.79.227.43:4157 , piece_id=42TVBZYTZ4O3AJ5FHBZMPBMWMEUIZDGY5NXYHAZ3TBKNHG6QCNRQ, seg_pos={0 0}, upload=533848]: Uploading piece... Dec 31 00:37:21.806866076 (609h8m21.677949121s) [noise=true , address=145.79.227.19:4079 , piece_id=W2AGJA3O2XARCZEI5USBHZE6EIWIDAFU24WKPCUHQII65YQ2VAKA, seg_pos={0 0}, upload=533848]: Uploading piece... Dec 31 00:37:21.806964183 (609h8m21.678047228s) [noise=true , address=77.90.31.13:4629 , piece_id=WAZYSDAN5MSE3C3UGJ5BQ7GDOPLPAOB7YQTHYUTIE7OZWRIVEARA, seg_pos={0 0}, upload=533848]: Uploading piece... Dec 31 00:37:21.807173682 (609h8m21.678256717s) [noise=true , address=145.79.227.10:4402 , piece_id=UJXD3JBNBCKBK2H4OAW4KBCFXMZMP3MHDKYZOYFYSKCAJDWIU4YQ, seg_pos={0 0}, upload=533848]: Uploading piece... Dec 31 00:37:21.807465367 (609h8m21.678548402s) [noise=true , address=45.91.116.9:4262 , piece_id=6FNAPYOYWGGJYCTPXCMWI7L325YOIP6RDE7UZGJUH5XZICAA4CTQ, seg_pos={0 0}, upload=533848]: Uploading piece... Dec 31 00:37:21.807912939 (609h8m21.678995984s) [noise=true , address=77.90.31.14:4663 , piece_id=BEL6CMHHCPH6PJBFQBY2P22GI2DSHLU6C2FI3KREEKFXVESBJROQ, seg_pos={0 0}, upload=533848]: Uploading piece... Dec 31 00:37:21.808755323 (609h8m21.679838368s) [noise=true , address=145.79.227.29:4942 , piece_id=3M3YMDPBFYAW2RZGTFMZLPVGDP7HY72DLHR2PALDI6FJFYDLHFAA, seg_pos={0 0}, upload=533848]: Uploading piece... Dec 31 00:37:23.506877615 (609h8m23.37796065s) [upload=533848]: Next returned nil segment Dec 31 00:37:23.506893356 (609h8m23.377976401s) [upload=533848]: Waiting for error group managing segments... Dec 31 00:37:23.787196661 (609h8m23.658279696s) [noise=true , address=145.79.227.19:4079 , piece_id=W2AGJA3O2XARCZEI5USBHZE6EIWIDAFU24WKPCUHQII65YQ2VAKA, seg_pos={0 0}, upload=533848]: Done uploading piece. err: <nil> Dec 31 00:37:23.794707001 (609h8m23.665790046s) [noise=true , address=145.79.227.31:4797 , piece_id=IZMS3LM7IG3RBXIMQTRDCINIQBQ4MNQM4XXDSJUTEJBBVTMSEZEA, seg_pos={0 0}, upload=533848]: Done uploading piece. err: <nil> Dec 31 00:37:23.802772670 (609h8m23.673855705s) [noise=true , address=145.79.227.10:4402 , piece_id=UJXD3JBNBCKBK2H4OAW4KBCFXMZMP3MHDKYZOYFYSKCAJDWIU4YQ, seg_pos={0 0}, upload=533848]: Done uploading piece. err: <nil> Dec 31 00:37:23.809582226 (609h8m23.680665261s) [noise=true , address=145.79.227.11:4732 , piece_id=NXZGC73AGULETYQ2EMVQ7XM36TNFNECSTUGTI3VMCPXJGGY2ALQA, seg_pos={0 0}, upload=533848]: Done uploading piece. err: <nil> Dec 31 00:37:23.879984451 (609h8m23.751067486s) [noise=true , address=145.79.227.41:4086 , piece_id=KCXGGAOKZNWFLLBV7I6P7LHVNM6DFRN5DITCURNMWBA2VE7CKVHA, seg_pos={0 0}, upload=533848]: Done uploading piece. err: <nil> Dec 31 00:37:23.982919423 (609h8m23.854002458s) [noise=true , address=77.90.31.13:4629 , piece_id=WAZYSDAN5MSE3C3UGJ5BQ7GDOPLPAOB7YQTHYUTIE7OZWRIVEARA, seg_pos={0 0}, upload=533848]: Done uploading piece. err: <nil> Dec 31 00:37:24.025039846 (609h8m23.896122881s) [noise=true , address=77.90.31.18:4805 , piece_id=QRFMMAPSLAYNXMN5JG2PKJ5DZUGPOXUSDBYRBGCYR34WT63APUOA, seg_pos={0 0}, upload=533848]: Done uploading piece. err: <nil> Dec 31 00:37:24.081686509 (609h8m23.952769695s) [noise=true , address=45.91.116.8:4045 , piece_id=RTVLMBX2SLOK23HMAWEA4G6WHEEZEBH6XQRKKRK6WD5GD6BDW7NA, seg_pos={0 0}, upload=533848]: Done uploading piece. err: <nil> Dec 31 00:37:24.116560253 (609h8m23.987643288s) [noise=true , address=45.91.116.5:4299 , piece_id=2GPR5IUAX2AHYXUSSLGXOOU6C3PYEPIY6P4B67GBTMIE5X4FFVHQ, seg_pos={0 0}, upload=533848]: Done uploading piece. err: <nil> Dec 31 00:37:24.152081901 (609h8m24.023164936s) [noise=true , address=45.91.116.10:4081 , piece_id=6GJLWPKDJL6TUXLQLGS62F7YG6PPZ42O3VQT6KQFNTTBOZVSFDGQ, seg_pos={0 0}, upload=533848]: Done uploading piece. err: <nil> Dec 31 00:37:24.167484239 (609h8m24.038567284s) [noise=true , address=77.90.31.14:4663 , piece_id=BEL6CMHHCPH6PJBFQBY2P22GI2DSHLU6C2FI3KREEKFXVESBJROQ, seg_pos={0 0}, upload=533848]: Done uploading piece. err: <nil> Dec 31 00:37:24.199729909 (609h8m24.070812944s) [noise=true , address=45.91.116.9:4262 , piece_id=6FNAPYOYWGGJYCTPXCMWI7L325YOIP6RDE7UZGJUH5XZICAA4CTQ, seg_pos={0 0}, upload=533848]: Done uploading piece. err: <nil> Dec 31 00:37:24.728312881 (609h8m24.599395926s) [noise=true , address=77.90.31.21:4878 , piece_id=T6MNCWRE4CB45ARESCQV3UCXA6FHSBVSC6SFJCRA4FOKL4QJ7Y2A, seg_pos={0 0}, upload=533848]: Done uploading piece. err: <nil> Dec 31 00:37:24.728331858 (609h8m24.599414893s) [seg_pos={0 0}, upload=533848]: Segment reached optimal threshold of 13 pieces. Dec 31 00:37:24.728520977 (609h8m24.599604013s) [noise=true , address=145.79.227.43:4157 , piece_id=42TVBZYTZ4O3AJ5FHBZMPBMWMEUIZDGY5NXYHAZ3TBKNHG6QCNRQ, seg_pos={0 0}, upload=533848]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZrzADk2mKG4R9tjxUKjQeJvb4WmfjLksGFyvstHRfYbBC7XNz): protocol: expected piece hash; context canceled; EOF Dec 31 00:37:24.728527470 (609h8m24.599610505s) [noise=true , address=145.79.227.43:4157 , piece_id=42TVBZYTZ4O3AJ5FHBZMPBMWMEUIZDGY5NXYHAZ3TBKNHG6QCNRQ, seg_pos={0 0}, upload=533848]: Long tail cancellation due to optimal threshold Dec 31 00:37:24.728572045 (609h8m24.59965508s) [noise=true , address=145.79.227.29:4942 , piece_id=3M3YMDPBFYAW2RZGTFMZLPVGDP7HY72DLHR2PALDI6FJFYDLHFAA, seg_pos={0 0}, upload=533848]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17o881pUMuG7K2dv83k7RkrJCrmM7kgUiWesgam4GEw21KVUn7): protocol: expected piece hash; context canceled; EOF Dec 31 00:37:24.728580772 (609h8m24.599663807s) [noise=true , address=145.79.227.29:4942 , piece_id=3M3YMDPBFYAW2RZGTFMZLPVGDP7HY72DLHR2PALDI6FJFYDLHFAA, seg_pos={0 0}, upload=533848]: Long tail cancellation due to optimal threshold Dec 31 00:37:24.728718213 (609h8m24.599801258s) [noise=true , address=45.91.116.6:4975 , piece_id=AC5RI6DJ3L5QPO22FF3J3KQW2UEQVC5QBD247E35JIMIIQ5OQP4A, seg_pos={0 0}, upload=533848]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Dec 31 00:37:24.728724195 (609h8m24.59980723s) [noise=true , address=45.91.116.6:4975 , piece_id=AC5RI6DJ3L5QPO22FF3J3KQW2UEQVC5QBD247E35JIMIIQ5OQP4A, seg_pos={0 0}, upload=533848]: Long tail cancellation due to optimal threshold Dec 31 00:37:24.728742089 (609h8m24.599825124s) [seg_pos={0 0}, upload=533848]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 00:37:24.728824065 (609h8m24.5999071s) [upload=533848]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 00:37:24.728828854 (609h8m24.599911889s) [upload=533848]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 00:37:25.041591605 (609h8m24.91267466s) [upload=533848]: Done uploading segments... Dec 31 00:37:25.041615270 (609h8m24.912698315s) [upload=533848]: Upload finished. err: <nil> Duration: 30.00257802s core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 5.069961487s trace: https://jaeger.dev.storj.io/trace/63d344eb9b98ed45 FAIL (2025-12-31 00:37:33 UTC) orbiter warning: operation: DeleteObject uplink: metaclient: context deadline exceeded Duration: 5.069988849s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 15.167032278s trace: https://jaeger.dev.storj.io/trace/67f6164cdd15ab81 FAIL (2025-12-31 00:37:24 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== 31 00:33:25.026367244 (609h4m24.897450289s) [noise=true , address=145.79.227.2:4128 , piece_id=DW37Q32FFAZ37KHMOII6XGCJVDO2XTKED5TLOKG65JULWTXJIQEA, seg_pos={0 0}, upload=533792]: Uploading piece... Dec 31 00:33:25.026878247 (609h4m24.897961292s) [noise=true , address=145.79.227.44:4217 , piece_id=GW44B36EHWVCRLEM6BA5MXC3ITRNCCJATW2RXSYTNC7K7WRO2IDQ, seg_pos={0 0}, upload=533792]: Uploading piece... Dec 31 00:33:25.027303136 (609h4m24.898386171s) [noise=true , address=45.91.116.13:4161 , piece_id=K4CTNQGHL5S2GOARW2F42REGZ4QVWA2RKUNULONYX3VPVHAGONUQ, seg_pos={0 0}, upload=533792]: Uploading piece... Dec 31 00:33:25.027901605 (609h4m24.89898465s) [noise=true , address=77.90.31.13:4632 , piece_id=5KUEGWEJV5NJE2EX3S5S2G7TS5TK23ZQXV3WPA4LLHO65QJRVW6Q, seg_pos={0 0}, upload=533792]: Uploading piece... Dec 31 00:33:25.028257373 (609h4m24.899340408s) [noise=true , address=145.79.227.30:4769 , piece_id=P2ANRKDKCOTOMCZZMDIH2YRVI366UOHWO3LXV7INEDB2GW35Y52A, seg_pos={0 0}, upload=533792]: Uploading piece... Dec 31 00:33:25.028623710 (609h4m24.899706745s) [noise=true , address=145.79.227.34:4930 , piece_id=L3AVRBWPDFTZANZKO4TGITCA3WHBECKEQ2BGMLFQJYN4CR7FSADA, seg_pos={0 0}, upload=533792]: Uploading piece... Dec 31 00:33:25.028952256 (609h4m24.900035291s) [noise=true , address=145.79.227.25:4630 , piece_id=5BVJWWAW2QUQABWWZCD4EW54ZVH3K5RJNM5EXJJUGXI33O6CC24A, seg_pos={0 0}, upload=533792]: Uploading piece... Dec 31 00:33:25.029314766 (609h4m24.900397801s) [noise=true , address=145.79.227.29:4971 , piece_id=EIY6YRG5ZRMEW7IWE5X25RCTGNCU3HQ27O7Q434OAZ4EI335HU4A, seg_pos={0 0}, upload=533792]: Uploading piece... Dec 31 00:33:25.029659533 (609h4m24.900742578s) [noise=true , address=145.79.227.6:4283 , piece_id=MXSG7TVTQBR2AFONVXYWQB4DBAQEEXW226FDC6SSVG5TTWGPAWFQ, seg_pos={0 0}, upload=533792]: Uploading piece... Dec 31 00:33:25.030576118 (609h4m24.901659153s) [noise=true , address=77.90.31.18:4835 , piece_id=GRMJTN6ZDDYITHJWX5UGIAGLCOW6AKADHTK6UCVGRB437TUTDFGA, seg_pos={0 0}, upload=533792]: Uploading piece... Dec 31 00:33:25.031413452 (609h4m24.902496487s) [noise=true , address=64.49.13.155:4913 , piece_id=E4WJCMYJVURPHENWEWR46KOAUAOZU7FW7WLC53DAU44BE7R3CHMQ, seg_pos={0 0}, upload=533792]: Uploading piece... Dec 31 00:33:25.031987204 (609h4m24.903070239s) [noise=true , address=145.79.227.49:4370 , piece_id=52YCNJY7BLJ3N3HDWBZCM7PP2ZPHHBS4GV4QS4WM2AWSLYKB5BBA, seg_pos={0 0}, upload=533792]: Uploading piece... Dec 31 00:33:25.032849947 (609h4m24.903932993s) [noise=true , address=145.79.227.4:4189 , piece_id=ZZF4DSIPQBWKBD45UR2J5P75U5VVMTVB54NEX5QKZV4K2WJZ43BQ, seg_pos={0 0}, upload=533792]: Uploading piece... Dec 31 00:33:25.034222320 (609h4m24.905305355s) [noise=true , address=145.79.227.49:4353 , piece_id=MVTAW6QPJY2LRNEGISHNDP7AC5ZIIUR5N5BBZRFO5I6VYJU7UR3Q, seg_pos={0 0}, upload=533792]: Uploading piece... Dec 31 00:33:25.035359376 (609h4m24.906442411s) [noise=true , address=145.79.227.35:4494 , piece_id=EY2OJLQIF4CH7DQWJWXSBJMQI37NAXSBLQP4ZPMU3IBRVLWJVYWA, seg_pos={0 0}, upload=533792]: Uploading piece... Dec 31 00:33:25.035852344 (609h4m24.906935379s) [noise=true , address=145.79.227.33:4873 , piece_id=2DSEGSC5UWL76NEXSE7WXOGSQNRNXALAINY52TLQHB57LTS2R6ZQ, seg_pos={0 0}, upload=533792]: Uploading piece... Dec 31 00:33:26.491429069 (609h4m26.362512104s) [upload=533792]: Next returned nil segment Dec 31 00:33:26.491441904 (609h4m26.362524939s) [upload=533792]: Waiting for error group managing segments... Dec 31 00:33:26.876786140 (609h4m26.747869175s) [noise=true , address=145.79.227.6:4283 , piece_id=MXSG7TVTQBR2AFONVXYWQB4DBAQEEXW226FDC6SSVG5TTWGPAWFQ, seg_pos={0 0}, upload=533792]: Done uploading piece. err: <nil> Dec 31 00:33:26.895923271 (609h4m26.767006316s) [noise=true , address=145.79.227.35:4494 , piece_id=EY2OJLQIF4CH7DQWJWXSBJMQI37NAXSBLQP4ZPMU3IBRVLWJVYWA, seg_pos={0 0}, upload=533792]: Done uploading piece. err: <nil> Dec 31 00:33:26.903106888 (609h4m26.774189923s) [noise=true , address=145.79.227.33:4873 , piece_id=2DSEGSC5UWL76NEXSE7WXOGSQNRNXALAINY52TLQHB57LTS2R6ZQ, seg_pos={0 0}, upload=533792]: Done uploading piece. err: <nil> Dec 31 00:33:26.912484285 (609h4m26.78356733s) [noise=true , address=145.79.227.29:4971 , piece_id=EIY6YRG5ZRMEW7IWE5X25RCTGNCU3HQ27O7Q434OAZ4EI335HU4A, seg_pos={0 0}, upload=533792]: Done uploading piece. err: <nil> Dec 31 00:33:27.004280888 (609h4m26.875363933s) [noise=true , address=145.79.227.21:4495 , piece_id=YWETKWGBGTY3LALK4DXWSKUTM2TFEYB4N5A3KNKW346V3F5XRN2Q, seg_pos={0 0}, upload=533792]: Done uploading piece. err: <nil> Dec 31 00:33:27.118759862 (609h4m26.989842897s) [noise=true , address=77.90.31.13:4632 , piece_id=5KUEGWEJV5NJE2EX3S5S2G7TS5TK23ZQXV3WPA4LLHO65QJRVW6Q, seg_pos={0 0}, upload=533792]: Done uploading piece. err: <nil> Dec 31 00:33:27.225324635 (609h4m27.09640767s) [noise=true , address=45.91.116.13:4161 , piece_id=K4CTNQGHL5S2GOARW2F42REGZ4QVWA2RKUNULONYX3VPVHAGONUQ, seg_pos={0 0}, upload=533792]: Done uploading piece. err: <nil> Dec 31 00:33:27.347765488 (609h4m27.218848533s) [noise=true , address=45.91.116.12:4248 , piece_id=QXKIUKBDX2GSJ2EQPALFCLPPLNWBPL3GRAEHLPJNSWUDIMU7WHYA, seg_pos={0 0}, upload=533792]: Done uploading piece. err: <nil> Dec 31 00:33:27.789508217 (609h4m27.660591252s) [noise=true , address=145.79.227.44:4217 , piece_id=GW44B36EHWVCRLEM6BA5MXC3ITRNCCJATW2RXSYTNC7K7WRO2IDQ, seg_pos={0 0}, upload=533792]: Done uploading piece. err: <nil> Dec 31 00:33:27.821042021 (609h4m27.692125056s) [noise=true , address=145.79.227.49:4353 , piece_id=MVTAW6QPJY2LRNEGISHNDP7AC5ZIIUR5N5BBZRFO5I6VYJU7UR3Q, seg_pos={0 0}, upload=533792]: Done uploading piece. err: <nil> Dec 31 00:33:27.828648224 (609h4m27.699731259s) [noise=true , address=145.79.227.49:4370 , piece_id=52YCNJY7BLJ3N3HDWBZCM7PP2ZPHHBS4GV4QS4WM2AWSLYKB5BBA, seg_pos={0 0}, upload=533792]: Done uploading piece. err: <nil> Dec 31 00:33:27.840271497 (609h4m27.711354542s) [noise=true , address=145.79.227.30:4769 , piece_id=P2ANRKDKCOTOMCZZMDIH2YRVI366UOHWO3LXV7INEDB2GW35Y52A, seg_pos={0 0}, upload=533792]: Done uploading piece. err: <nil> Dec 31 00:33:27.844064569 (609h4m27.715147614s) [noise=true , address=145.79.227.34:4930 , piece_id=L3AVRBWPDFTZANZKO4TGITCA3WHBECKEQ2BGMLFQJYN4CR7FSADA, seg_pos={0 0}, upload=533792]: Done uploading piece. err: <nil> Dec 31 00:33:27.867248213 (609h4m27.738331248s) [noise=true , address=77.90.31.18:4835 , piece_id=GRMJTN6ZDDYITHJWX5UGIAGLCOW6AKADHTK6UCVGRB437TUTDFGA, seg_pos={0 0}, upload=533792]: Done uploading piece. err: <nil> Dec 31 00:33:27.867270586 (609h4m27.738353621s) [seg_pos={0 0}, upload=533792]: Segment reached optimal threshold of 14 pieces. Dec 31 00:33:27.867476859 (609h4m27.738559904s) [noise=true , address=64.49.13.155:4913 , piece_id=E4WJCMYJVURPHENWEWR46KOAUAOZU7FW7WLC53DAU44BE7R3CHMQ, seg_pos={0 0}, upload=533792]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oyYUnWDyFo1npaKpBxMG5K31oUzJnTpcU1XHFmKux3MCVUPZV): protocol: expected piece hash; context canceled; EOF Dec 31 00:33:27.867485555 (609h4m27.73856859s) [noise=true , address=64.49.13.155:4913 , piece_id=E4WJCMYJVURPHENWEWR46KOAUAOZU7FW7WLC53DAU44BE7R3CHMQ, seg_pos={0 0}, upload=533792]: Long tail cancellation due to optimal threshold Dec 31 00:33:27.867516204 (609h4m27.738599249s) [noise=true , address=145.79.227.25:4630 , piece_id=5BVJWWAW2QUQABWWZCD4EW54ZVH3K5RJNM5EXJJUGXI33O6CC24A, seg_pos={0 0}, upload=533792]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12aETKgTzKw8fyYqbeDA2JzfxZnWn12xUPYK1eRt6TvBPPdRJDp): protocol: expected piece hash; context canceled; EOF Dec 31 00:33:27.867526553 (609h4m27.738609588s) [noise=true , address=145.79.227.25:4630 , piece_id=5BVJWWAW2QUQABWWZCD4EW54ZVH3K5RJNM5EXJJUGXI33O6CC24A, seg_pos={0 0}, upload=533792]: Long tail cancellation due to optimal threshold Dec 31 00:33:27.867714432 (609h4m27.738797477s) [noise=true , address=145.79.227.4:4189 , piece_id=ZZF4DSIPQBWKBD45UR2J5P75U5VVMTVB54NEX5QKZV4K2WJZ43BQ, seg_pos={0 0}, upload=533792]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jH3757i2F3q18XSpoXznsysZPaXmEnqynpLPDupgRfxJrTgKv): protocol: write tcp 188.166.225.170:57528->145.79.227.4:4189: use of closed network connection; write tcp 188.166.225.170:57528->145.79.227.4:4189: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57528->145.79.227.4:4189: use of closed network connection Dec 31 00:33:27.867723689 (609h4m27.738806724s) [noise=true , address=145.79.227.4:4189 , piece_id=ZZF4DSIPQBWKBD45UR2J5P75U5VVMTVB54NEX5QKZV4K2WJZ43BQ, seg_pos={0 0}, upload=533792]: Long tail cancellation due to optimal threshold Dec 31 00:33:27.867744699 (609h4m27.738827734s) [noise=true , address=145.79.227.2:4128 , piece_id=DW37Q32FFAZ37KHMOII6XGCJVDO2XTKED5TLOKG65JULWTXJIQEA, seg_pos={0 0}, upload=533792]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1wABkAtAjoYkKyNoAiaVCxSvY6WgvHvAYFPCjpNojtpiYiGtDS): protocol: expected piece hash; context canceled; EOF Dec 31 00:33:27.867753616 (609h4m27.738836651s) [noise=true , address=145.79.227.2:4128 , piece_id=DW37Q32FFAZ37KHMOII6XGCJVDO2XTKED5TLOKG65JULWTXJIQEA, seg_pos={0 0}, upload=533792]: Long tail cancellation due to optimal threshold Dec 31 00:33:27.867777462 (609h4m27.738860507s) [seg_pos={0 0}, upload=533792]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 31 00:33:27.867853687 (609h4m27.738936722s) [upload=533792]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 00:33:27.867859989 (609h4m27.738943024s) [upload=533792]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 00:33:28.167335511 (609h4m28.038418546s) [upload=533792]: Done uploading segments... Dec 31 00:33:28.167355899 (609h4m28.038438934s) [upload=533792]: Upload finished. err: <nil> Duration: 15.167061934s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [.................X...X...........X........X..X....XX.XXX....] duration: 6.060794664s trace: https://jaeger.dev.storj.io/trace/113f1a95e44d0023 FAIL (2025-12-31 00:37:01 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, Piece ID: S4TGVS42HN5MRIBKUR654AK2WOERFXKWPZTJZVK44UR7IN5FN55Q) context deadline exceeded; piecestore: (Node ID: 1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, Piece ID: UICQMYJTR5YEKUD7M4OJRED57CI2G237GT3EMYPGMWKYSCE4GRQA) context deadline exceeded ======LOGS====== Dec 31 00:33:43.715936146 (609h4m43.587019181s) [upload=533798]: Starting upload Dec 31 00:33:43.715992443 (609h4m43.587075478s) [upload=533798]: Done starting upload Dec 31 00:33:43.716132921 (609h4m43.587215976s) [upload=533798]: Uploading segments... Dec 31 00:33:43.716383137 (609h4m43.587466172s) [upload=533798]: Got next segment. Inline: false Dec 31 00:33:43.716430236 (609h4m43.587513281s) [upload=533798]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 00:33:43.716456406 (609h4m43.587539441s) [upload=533798]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 00:33:43.961837344 (609h4m43.832920389s) [seg_pos={0 0}, upload=533798]: Begin upload segment... Dec 31 00:33:43.961889664 (609h4m43.832972709s) [seg_pos={0 0}, upload=533798]: Done begin upload segment. Dec 31 00:33:43.961925643 (609h4m43.833008678s) [noise=true , address=145.79.227.23:4547 , piece_id=TRELDHWOXL7AXM4POWMEWGUBBHN47JXSBQV5NUI4EEPMLIEPJLVQ, seg_pos={0 0}, upload=533798]: Uploading piece... Dec 31 00:33:43.961935040 (609h4m43.833018075s) [noise=true , address=145.79.227.8:4329 , piece_id=CUZF4JU4IPCH6DXOEWOOHGCUTHDNAX2LOKLEXX2GFW5MAFAJ5BCA, seg_pos={0 0}, upload=533798]: Uploading piece... Dec 31 00:33:43.962101387 (609h4m43.833184422s) [noise=true , address=145.79.227.37:4566 , piece_id=YRRPITFA3W2PA7L4KRJ6OYD5SEBCMAYVOPZA4R56D23XEEQZGHFQ, seg_pos={0 0}, upload=533798]: Uploading piece... Dec 31 00:33:43.962628120 (609h4m43.833711155s) [noise=true , address=145.79.227.2:4113 , piece_id=ZGKPJDMEGSRL6GKBFCF7CZVKCJM5PW4G6Z7LF3N7UEA3E4WMNFRA, seg_pos={0 0}, upload=533798]: Uploading piece... Dec 31 00:33:43.963204478 (609h4m43.834287513s) [noise=true , address=77.90.31.21:4877 , piece_id=S4TGVS42HN5MRIBKUR654AK2WOERFXKWPZTJZVK44UR7IN5FN55Q, seg_pos={0 0}, upload=533798]: Uploading piece... Dec 31 00:33:43.963688299 (609h4m43.834771334s) [noise=true , address=45.91.116.13:4156 , piece_id=IWYIJHXM5SF2HU337U7A3YU4ITYGZDJAKQLNKKUIZGNKFVVCQQJQ, seg_pos={0 0}, upload=533798]: Uploading piece... Dec 31 00:33:43.964191076 (609h4m43.835274111s) [noise=true , address=77.90.31.13:4627 , piece_id=UICQMYJTR5YEKUD7M4OJRED57CI2G237GT3EMYPGMWKYSCE4GRQA, seg_pos={0 0}, upload=533798]: Uploading piece... Dec 31 00:33:43.964608601 (609h4m43.835691646s) [noise=true , address=145.79.227.4:4187 , piece_id=TNOJCADMYMM2ZXMSWYWRWGJTNWVHX2CAMUMXY236S63B4MLVE43A, seg_pos={0 0}, upload=533798]: Uploading piece... Dec 31 00:33:43.965019284 (609h4m43.836102319s) [noise=true , address=145.79.227.19:4078 , piece_id=RIQ3XPKHYS25Z55CGZAMLGACVBWOJBCR5EGYOZPSCS4KZMCYNTSA, seg_pos={0 0}, upload=533798]: Uploading piece... Dec 31 00:33:43.966236792 (609h4m43.837319837s) [noise=true , address=145.79.227.50:4374 , piece_id=BLDWCE4SFJQA7ZWGDDZ5XYBBMH432M56LB7ZF3LPDB4WAVBR4GBA, seg_pos={0 0}, upload=533798]: Uploading piece... Dec 31 00:33:43.967461544 (609h4m43.838544589s) [noise=true , address=145.79.227.27:4691 , piece_id=AED64GHR5KJD75QSG3W2Q54Y4H2N3DCLK4IYWF3JLR7GXOKDOD5A, seg_pos={0 0}, upload=533798]: Uploading piece... Dec 31 00:33:43.967896762 (609h4m43.838979807s) [noise=true , address=145.79.227.36:4517 , piece_id=ZOMEYIPAPM24XKFQHAQYDZ56ZDCPAILUBC7BYS6NFHVQYKOCBCBA, seg_pos={0 0}, upload=533798]: Uploading piece... Dec 31 00:33:43.969181198 (609h4m43.840264233s) [noise=true , address=45.91.116.4:4937 , piece_id=TKRNJXBY4X2XZEPEAJAQNVHGB7EO4MWG65JXD4MMJN4AMRYNG3DA, seg_pos={0 0}, upload=533798]: Uploading piece... Dec 31 00:33:43.974310605 (609h4m43.84539366s) [noise=true , address=45.91.116.6:4976 , piece_id=BST4NLF5ZD765L5YIUXDSAXYIK2K7Z3U65JJLKGQ3KWBTCPZFRZQ, seg_pos={0 0}, upload=533798]: Uploading piece... Dec 31 00:33:43.974435924 (609h4m43.845518969s) [noise=true , address=145.79.227.14:4839 , piece_id=GY4GTCAXYTQ6IYNRZIICFQRPRTSP3HWGKITMNPMWPVGDC724Q6AQ, seg_pos={0 0}, upload=533798]: Uploading piece... Dec 31 00:33:43.975830920 (609h4m43.846913955s) [noise=true , address=45.91.116.8:4046 , piece_id=ROT7BI4CVLZTQ4OLSITC7GA64IF7QFSFFCLAN3XC4IZGYNHFMHXQ, seg_pos={0 0}, upload=533798]: Uploading piece... Dec 31 00:33:45.660804749 (609h4m45.531887784s) [upload=533798]: Next returned nil segment Dec 31 00:33:45.660820368 (609h4m45.531903413s) [upload=533798]: Waiting for error group managing segments... Dec 31 00:33:45.951050435 (609h4m45.82213347s) [noise=true , address=145.79.227.4:4187 , piece_id=TNOJCADMYMM2ZXMSWYWRWGJTNWVHX2CAMUMXY236S63B4MLVE43A, seg_pos={0 0}, upload=533798]: Done uploading piece. err: <nil> Dec 31 00:33:45.952806208 (609h4m45.823889254s) [noise=true , address=145.79.227.27:4691 , piece_id=AED64GHR5KJD75QSG3W2Q54Y4H2N3DCLK4IYWF3JLR7GXOKDOD5A, seg_pos={0 0}, upload=533798]: Done uploading piece. err: <nil> Dec 31 00:33:45.968805193 (609h4m45.839888228s) [noise=true , address=145.79.227.2:4113 , piece_id=ZGKPJDMEGSRL6GKBFCF7CZVKCJM5PW4G6Z7LF3N7UEA3E4WMNFRA, seg_pos={0 0}, upload=533798]: Done uploading piece. err: <nil> Dec 31 00:33:45.970513926 (609h4m45.841596961s) [noise=true , address=145.79.227.23:4547 , piece_id=TRELDHWOXL7AXM4POWMEWGUBBHN47JXSBQV5NUI4EEPMLIEPJLVQ, seg_pos={0 0}, upload=533798]: Done uploading piece. err: <nil> Dec 31 00:33:45.973565597 (609h4m45.844648632s) [noise=true , address=145.79.227.36:4517 , piece_id=ZOMEYIPAPM24XKFQHAQYDZ56ZDCPAILUBC7BYS6NFHVQYKOCBCBA, seg_pos={0 0}, upload=533798]: Done uploading piece. err: <nil> Dec 31 00:33:45.974458116 (609h4m45.845541152s) [noise=true , address=145.79.227.8:4329 , piece_id=CUZF4JU4IPCH6DXOEWOOHGCUTHDNAX2LOKLEXX2GFW5MAFAJ5BCA, seg_pos={0 0}, upload=533798]: Done uploading piece. err: <nil> Dec 31 00:33:45.974561093 (609h4m45.845644138s) [noise=true , address=145.79.227.37:4566 , piece_id=YRRPITFA3W2PA7L4KRJ6OYD5SEBCMAYVOPZA4R56D23XEEQZGHFQ, seg_pos={0 0}, upload=533798]: Done uploading piece. err: <nil> Dec 31 00:33:45.983584214 (609h4m45.854667259s) [noise=true , address=145.79.227.50:4374 , piece_id=BLDWCE4SFJQA7ZWGDDZ5XYBBMH432M56LB7ZF3LPDB4WAVBR4GBA, seg_pos={0 0}, upload=533798]: Done uploading piece. err: <nil> Dec 31 00:33:45.999281684 (609h4m45.870364729s) [noise=true , address=145.79.227.14:4839 , piece_id=GY4GTCAXYTQ6IYNRZIICFQRPRTSP3HWGKITMNPMWPVGDC724Q6AQ, seg_pos={0 0}, upload=533798]: Done uploading piece. err: <nil> Dec 31 00:33:46.026097925 (609h4m45.89718097s) [noise=true , address=145.79.227.19:4078 , piece_id=RIQ3XPKHYS25Z55CGZAMLGACVBWOJBCR5EGYOZPSCS4KZMCYNTSA, seg_pos={0 0}, upload=533798]: Done uploading piece. err: <nil> Dec 31 00:33:46.119472633 (609h4m45.990555678s) [noise=true , address=77.90.31.21:4877 , piece_id=S4TGVS42HN5MRIBKUR654AK2WOERFXKWPZTJZVK44UR7IN5FN55Q, seg_pos={0 0}, upload=533798]: Done uploading piece. err: <nil> Dec 31 00:33:46.223788906 (609h4m46.094871941s) [noise=true , address=77.90.31.13:4627 , piece_id=UICQMYJTR5YEKUD7M4OJRED57CI2G237GT3EMYPGMWKYSCE4GRQA, seg_pos={0 0}, upload=533798]: Done uploading piece. err: <nil> Dec 31 00:33:46.307787982 (609h4m46.178871017s) [noise=true , address=45.91.116.13:4156 , piece_id=IWYIJHXM5SF2HU337U7A3YU4ITYGZDJAKQLNKKUIZGNKFVVCQQJQ, seg_pos={0 0}, upload=533798]: Done uploading piece. err: <nil> Dec 31 00:33:46.307808380 (609h4m46.178891415s) [seg_pos={0 0}, upload=533798]: Segment reached optimal threshold of 13 pieces. Dec 31 00:33:46.308010665 (609h4m46.1790937s) [noise=true , address=45.91.116.8:4046 , piece_id=ROT7BI4CVLZTQ4OLSITC7GA64IF7QFSFFCLAN3XC4IZGYNHFMHXQ, seg_pos={0 0}, upload=533798]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H): protocol: expected piece hash; context canceled; EOF Dec 31 00:33:46.308018310 (609h4m46.179101345s) [noise=true , address=45.91.116.8:4046 , piece_id=ROT7BI4CVLZTQ4OLSITC7GA64IF7QFSFFCLAN3XC4IZGYNHFMHXQ, seg_pos={0 0}, upload=533798]: Long tail cancellation due to optimal threshold Dec 31 00:33:46.308129932 (609h4m46.179212967s) [noise=true , address=45.91.116.6:4976 , piece_id=BST4NLF5ZD765L5YIUXDSAXYIK2K7Z3U65JJLKGQ3KWBTCPZFRZQ, seg_pos={0 0}, upload=533798]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp): protocol: expected piece hash; context canceled; EOF Dec 31 00:33:46.308135213 (609h4m46.179218248s) [noise=true , address=45.91.116.6:4976 , piece_id=BST4NLF5ZD765L5YIUXDSAXYIK2K7Z3U65JJLKGQ3KWBTCPZFRZQ, seg_pos={0 0}, upload=533798]: Long tail cancellation due to optimal threshold Dec 31 00:33:46.308228550 (609h4m46.179311595s) [noise=true , address=45.91.116.4:4937 , piece_id=TKRNJXBY4X2XZEPEAJAQNVHGB7EO4MWG65JXD4MMJN4AMRYNG3DA, seg_pos={0 0}, upload=533798]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: expected piece hash; context canceled; EOF Dec 31 00:33:46.308234621 (609h4m46.179317656s) [noise=true , address=45.91.116.4:4937 , piece_id=TKRNJXBY4X2XZEPEAJAQNVHGB7EO4MWG65JXD4MMJN4AMRYNG3DA, seg_pos={0 0}, upload=533798]: Long tail cancellation due to optimal threshold Dec 31 00:33:46.308253347 (609h4m46.179336382s) [seg_pos={0 0}, upload=533798]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 00:33:46.308320215 (609h4m46.17940325s) [upload=533798]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 00:33:46.308324333 (609h4m46.179407368s) [upload=533798]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 00:33:46.594956498 (609h4m46.466039533s) [upload=533798]: Done uploading segments... Dec 31 00:33:46.594977478 (609h4m46.466060513s) [upload=533798]: Upload finished. err: <nil> Duration: 6.060853545s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 51.189525245s trace: https://jaeger.dev.storj.io/trace/67f6164cdd15ab81 FAIL (2025-12-31 00:36:51 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== 31 00:33:25.026367244 (609h4m24.897450289s) [noise=true , address=145.79.227.2:4128 , piece_id=DW37Q32FFAZ37KHMOII6XGCJVDO2XTKED5TLOKG65JULWTXJIQEA, seg_pos={0 0}, upload=533792]: Uploading piece... Dec 31 00:33:25.026878247 (609h4m24.897961292s) [noise=true , address=145.79.227.44:4217 , piece_id=GW44B36EHWVCRLEM6BA5MXC3ITRNCCJATW2RXSYTNC7K7WRO2IDQ, seg_pos={0 0}, upload=533792]: Uploading piece... Dec 31 00:33:25.027303136 (609h4m24.898386171s) [noise=true , address=45.91.116.13:4161 , piece_id=K4CTNQGHL5S2GOARW2F42REGZ4QVWA2RKUNULONYX3VPVHAGONUQ, seg_pos={0 0}, upload=533792]: Uploading piece... Dec 31 00:33:25.027901605 (609h4m24.89898465s) [noise=true , address=77.90.31.13:4632 , piece_id=5KUEGWEJV5NJE2EX3S5S2G7TS5TK23ZQXV3WPA4LLHO65QJRVW6Q, seg_pos={0 0}, upload=533792]: Uploading piece... Dec 31 00:33:25.028257373 (609h4m24.899340408s) [noise=true , address=145.79.227.30:4769 , piece_id=P2ANRKDKCOTOMCZZMDIH2YRVI366UOHWO3LXV7INEDB2GW35Y52A, seg_pos={0 0}, upload=533792]: Uploading piece... Dec 31 00:33:25.028623710 (609h4m24.899706745s) [noise=true , address=145.79.227.34:4930 , piece_id=L3AVRBWPDFTZANZKO4TGITCA3WHBECKEQ2BGMLFQJYN4CR7FSADA, seg_pos={0 0}, upload=533792]: Uploading piece... Dec 31 00:33:25.028952256 (609h4m24.900035291s) [noise=true , address=145.79.227.25:4630 , piece_id=5BVJWWAW2QUQABWWZCD4EW54ZVH3K5RJNM5EXJJUGXI33O6CC24A, seg_pos={0 0}, upload=533792]: Uploading piece... Dec 31 00:33:25.029314766 (609h4m24.900397801s) [noise=true , address=145.79.227.29:4971 , piece_id=EIY6YRG5ZRMEW7IWE5X25RCTGNCU3HQ27O7Q434OAZ4EI335HU4A, seg_pos={0 0}, upload=533792]: Uploading piece... Dec 31 00:33:25.029659533 (609h4m24.900742578s) [noise=true , address=145.79.227.6:4283 , piece_id=MXSG7TVTQBR2AFONVXYWQB4DBAQEEXW226FDC6SSVG5TTWGPAWFQ, seg_pos={0 0}, upload=533792]: Uploading piece... Dec 31 00:33:25.030576118 (609h4m24.901659153s) [noise=true , address=77.90.31.18:4835 , piece_id=GRMJTN6ZDDYITHJWX5UGIAGLCOW6AKADHTK6UCVGRB437TUTDFGA, seg_pos={0 0}, upload=533792]: Uploading piece... Dec 31 00:33:25.031413452 (609h4m24.902496487s) [noise=true , address=64.49.13.155:4913 , piece_id=E4WJCMYJVURPHENWEWR46KOAUAOZU7FW7WLC53DAU44BE7R3CHMQ, seg_pos={0 0}, upload=533792]: Uploading piece... Dec 31 00:33:25.031987204 (609h4m24.903070239s) [noise=true , address=145.79.227.49:4370 , piece_id=52YCNJY7BLJ3N3HDWBZCM7PP2ZPHHBS4GV4QS4WM2AWSLYKB5BBA, seg_pos={0 0}, upload=533792]: Uploading piece... Dec 31 00:33:25.032849947 (609h4m24.903932993s) [noise=true , address=145.79.227.4:4189 , piece_id=ZZF4DSIPQBWKBD45UR2J5P75U5VVMTVB54NEX5QKZV4K2WJZ43BQ, seg_pos={0 0}, upload=533792]: Uploading piece... Dec 31 00:33:25.034222320 (609h4m24.905305355s) [noise=true , address=145.79.227.49:4353 , piece_id=MVTAW6QPJY2LRNEGISHNDP7AC5ZIIUR5N5BBZRFO5I6VYJU7UR3Q, seg_pos={0 0}, upload=533792]: Uploading piece... Dec 31 00:33:25.035359376 (609h4m24.906442411s) [noise=true , address=145.79.227.35:4494 , piece_id=EY2OJLQIF4CH7DQWJWXSBJMQI37NAXSBLQP4ZPMU3IBRVLWJVYWA, seg_pos={0 0}, upload=533792]: Uploading piece... Dec 31 00:33:25.035852344 (609h4m24.906935379s) [noise=true , address=145.79.227.33:4873 , piece_id=2DSEGSC5UWL76NEXSE7WXOGSQNRNXALAINY52TLQHB57LTS2R6ZQ, seg_pos={0 0}, upload=533792]: Uploading piece... Dec 31 00:33:26.491429069 (609h4m26.362512104s) [upload=533792]: Next returned nil segment Dec 31 00:33:26.491441904 (609h4m26.362524939s) [upload=533792]: Waiting for error group managing segments... Dec 31 00:33:26.876786140 (609h4m26.747869175s) [noise=true , address=145.79.227.6:4283 , piece_id=MXSG7TVTQBR2AFONVXYWQB4DBAQEEXW226FDC6SSVG5TTWGPAWFQ, seg_pos={0 0}, upload=533792]: Done uploading piece. err: <nil> Dec 31 00:33:26.895923271 (609h4m26.767006316s) [noise=true , address=145.79.227.35:4494 , piece_id=EY2OJLQIF4CH7DQWJWXSBJMQI37NAXSBLQP4ZPMU3IBRVLWJVYWA, seg_pos={0 0}, upload=533792]: Done uploading piece. err: <nil> Dec 31 00:33:26.903106888 (609h4m26.774189923s) [noise=true , address=145.79.227.33:4873 , piece_id=2DSEGSC5UWL76NEXSE7WXOGSQNRNXALAINY52TLQHB57LTS2R6ZQ, seg_pos={0 0}, upload=533792]: Done uploading piece. err: <nil> Dec 31 00:33:26.912484285 (609h4m26.78356733s) [noise=true , address=145.79.227.29:4971 , piece_id=EIY6YRG5ZRMEW7IWE5X25RCTGNCU3HQ27O7Q434OAZ4EI335HU4A, seg_pos={0 0}, upload=533792]: Done uploading piece. err: <nil> Dec 31 00:33:27.004280888 (609h4m26.875363933s) [noise=true , address=145.79.227.21:4495 , piece_id=YWETKWGBGTY3LALK4DXWSKUTM2TFEYB4N5A3KNKW346V3F5XRN2Q, seg_pos={0 0}, upload=533792]: Done uploading piece. err: <nil> Dec 31 00:33:27.118759862 (609h4m26.989842897s) [noise=true , address=77.90.31.13:4632 , piece_id=5KUEGWEJV5NJE2EX3S5S2G7TS5TK23ZQXV3WPA4LLHO65QJRVW6Q, seg_pos={0 0}, upload=533792]: Done uploading piece. err: <nil> Dec 31 00:33:27.225324635 (609h4m27.09640767s) [noise=true , address=45.91.116.13:4161 , piece_id=K4CTNQGHL5S2GOARW2F42REGZ4QVWA2RKUNULONYX3VPVHAGONUQ, seg_pos={0 0}, upload=533792]: Done uploading piece. err: <nil> Dec 31 00:33:27.347765488 (609h4m27.218848533s) [noise=true , address=45.91.116.12:4248 , piece_id=QXKIUKBDX2GSJ2EQPALFCLPPLNWBPL3GRAEHLPJNSWUDIMU7WHYA, seg_pos={0 0}, upload=533792]: Done uploading piece. err: <nil> Dec 31 00:33:27.789508217 (609h4m27.660591252s) [noise=true , address=145.79.227.44:4217 , piece_id=GW44B36EHWVCRLEM6BA5MXC3ITRNCCJATW2RXSYTNC7K7WRO2IDQ, seg_pos={0 0}, upload=533792]: Done uploading piece. err: <nil> Dec 31 00:33:27.821042021 (609h4m27.692125056s) [noise=true , address=145.79.227.49:4353 , piece_id=MVTAW6QPJY2LRNEGISHNDP7AC5ZIIUR5N5BBZRFO5I6VYJU7UR3Q, seg_pos={0 0}, upload=533792]: Done uploading piece. err: <nil> Dec 31 00:33:27.828648224 (609h4m27.699731259s) [noise=true , address=145.79.227.49:4370 , piece_id=52YCNJY7BLJ3N3HDWBZCM7PP2ZPHHBS4GV4QS4WM2AWSLYKB5BBA, seg_pos={0 0}, upload=533792]: Done uploading piece. err: <nil> Dec 31 00:33:27.840271497 (609h4m27.711354542s) [noise=true , address=145.79.227.30:4769 , piece_id=P2ANRKDKCOTOMCZZMDIH2YRVI366UOHWO3LXV7INEDB2GW35Y52A, seg_pos={0 0}, upload=533792]: Done uploading piece. err: <nil> Dec 31 00:33:27.844064569 (609h4m27.715147614s) [noise=true , address=145.79.227.34:4930 , piece_id=L3AVRBWPDFTZANZKO4TGITCA3WHBECKEQ2BGMLFQJYN4CR7FSADA, seg_pos={0 0}, upload=533792]: Done uploading piece. err: <nil> Dec 31 00:33:27.867248213 (609h4m27.738331248s) [noise=true , address=77.90.31.18:4835 , piece_id=GRMJTN6ZDDYITHJWX5UGIAGLCOW6AKADHTK6UCVGRB437TUTDFGA, seg_pos={0 0}, upload=533792]: Done uploading piece. err: <nil> Dec 31 00:33:27.867270586 (609h4m27.738353621s) [seg_pos={0 0}, upload=533792]: Segment reached optimal threshold of 14 pieces. Dec 31 00:33:27.867476859 (609h4m27.738559904s) [noise=true , address=64.49.13.155:4913 , piece_id=E4WJCMYJVURPHENWEWR46KOAUAOZU7FW7WLC53DAU44BE7R3CHMQ, seg_pos={0 0}, upload=533792]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oyYUnWDyFo1npaKpBxMG5K31oUzJnTpcU1XHFmKux3MCVUPZV): protocol: expected piece hash; context canceled; EOF Dec 31 00:33:27.867485555 (609h4m27.73856859s) [noise=true , address=64.49.13.155:4913 , piece_id=E4WJCMYJVURPHENWEWR46KOAUAOZU7FW7WLC53DAU44BE7R3CHMQ, seg_pos={0 0}, upload=533792]: Long tail cancellation due to optimal threshold Dec 31 00:33:27.867516204 (609h4m27.738599249s) [noise=true , address=145.79.227.25:4630 , piece_id=5BVJWWAW2QUQABWWZCD4EW54ZVH3K5RJNM5EXJJUGXI33O6CC24A, seg_pos={0 0}, upload=533792]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12aETKgTzKw8fyYqbeDA2JzfxZnWn12xUPYK1eRt6TvBPPdRJDp): protocol: expected piece hash; context canceled; EOF Dec 31 00:33:27.867526553 (609h4m27.738609588s) [noise=true , address=145.79.227.25:4630 , piece_id=5BVJWWAW2QUQABWWZCD4EW54ZVH3K5RJNM5EXJJUGXI33O6CC24A, seg_pos={0 0}, upload=533792]: Long tail cancellation due to optimal threshold Dec 31 00:33:27.867714432 (609h4m27.738797477s) [noise=true , address=145.79.227.4:4189 , piece_id=ZZF4DSIPQBWKBD45UR2J5P75U5VVMTVB54NEX5QKZV4K2WJZ43BQ, seg_pos={0 0}, upload=533792]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jH3757i2F3q18XSpoXznsysZPaXmEnqynpLPDupgRfxJrTgKv): protocol: write tcp 188.166.225.170:57528->145.79.227.4:4189: use of closed network connection; write tcp 188.166.225.170:57528->145.79.227.4:4189: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57528->145.79.227.4:4189: use of closed network connection Dec 31 00:33:27.867723689 (609h4m27.738806724s) [noise=true , address=145.79.227.4:4189 , piece_id=ZZF4DSIPQBWKBD45UR2J5P75U5VVMTVB54NEX5QKZV4K2WJZ43BQ, seg_pos={0 0}, upload=533792]: Long tail cancellation due to optimal threshold Dec 31 00:33:27.867744699 (609h4m27.738827734s) [noise=true , address=145.79.227.2:4128 , piece_id=DW37Q32FFAZ37KHMOII6XGCJVDO2XTKED5TLOKG65JULWTXJIQEA, seg_pos={0 0}, upload=533792]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1wABkAtAjoYkKyNoAiaVCxSvY6WgvHvAYFPCjpNojtpiYiGtDS): protocol: expected piece hash; context canceled; EOF Dec 31 00:33:27.867753616 (609h4m27.738836651s) [noise=true , address=145.79.227.2:4128 , piece_id=DW37Q32FFAZ37KHMOII6XGCJVDO2XTKED5TLOKG65JULWTXJIQEA, seg_pos={0 0}, upload=533792]: Long tail cancellation due to optimal threshold Dec 31 00:33:27.867777462 (609h4m27.738860507s) [seg_pos={0 0}, upload=533792]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 31 00:33:27.867853687 (609h4m27.738936722s) [upload=533792]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 00:33:27.867859989 (609h4m27.738943024s) [upload=533792]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 00:33:28.167335511 (609h4m28.038418546s) [upload=533792]: Done uploading segments... Dec 31 00:33:28.167355899 (609h4m28.038438934s) [upload=533792]: Upload finished. err: <nil> Duration: 51.189551296s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 5.060764725s trace: https://jaeger.dev.storj.io/trace/63d344eb9b98ed45 FAIL (2025-12-31 00:36:51 UTC) orbiter warning: operation: DeleteObject uplink: metaclient: context deadline exceeded Duration: 5.06077825s core uplink eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 30.005257085s trace: https://jaeger.dev.storj.io/trace/63d344eb9b98ed45 FAIL (2025-12-31 00:36:46 UTC) orbiter warning: operation: DeleteObject uplink: metaclient: context deadline exceeded Duration: 30.005280301s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [..................X...X...........X........X..X....XX.XXX...] duration: 5.826736008s trace: https://jaeger.dev.storj.io/trace/113f1a95e44d0023 FAIL (2025-12-31 00:36:01 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, Piece ID: S4TGVS42HN5MRIBKUR654AK2WOERFXKWPZTJZVK44UR7IN5FN55Q) context deadline exceeded; piecestore: (Node ID: 1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, Piece ID: UICQMYJTR5YEKUD7M4OJRED57CI2G237GT3EMYPGMWKYSCE4GRQA) context deadline exceeded ======LOGS====== Dec 31 00:33:43.715936146 (609h4m43.587019181s) [upload=533798]: Starting upload Dec 31 00:33:43.715992443 (609h4m43.587075478s) [upload=533798]: Done starting upload Dec 31 00:33:43.716132921 (609h4m43.587215976s) [upload=533798]: Uploading segments... Dec 31 00:33:43.716383137 (609h4m43.587466172s) [upload=533798]: Got next segment. Inline: false Dec 31 00:33:43.716430236 (609h4m43.587513281s) [upload=533798]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 00:33:43.716456406 (609h4m43.587539441s) [upload=533798]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 00:33:43.961837344 (609h4m43.832920389s) [seg_pos={0 0}, upload=533798]: Begin upload segment... Dec 31 00:33:43.961889664 (609h4m43.832972709s) [seg_pos={0 0}, upload=533798]: Done begin upload segment. Dec 31 00:33:43.961925643 (609h4m43.833008678s) [noise=true , address=145.79.227.23:4547 , piece_id=TRELDHWOXL7AXM4POWMEWGUBBHN47JXSBQV5NUI4EEPMLIEPJLVQ, seg_pos={0 0}, upload=533798]: Uploading piece... Dec 31 00:33:43.961935040 (609h4m43.833018075s) [noise=true , address=145.79.227.8:4329 , piece_id=CUZF4JU4IPCH6DXOEWOOHGCUTHDNAX2LOKLEXX2GFW5MAFAJ5BCA, seg_pos={0 0}, upload=533798]: Uploading piece... Dec 31 00:33:43.962101387 (609h4m43.833184422s) [noise=true , address=145.79.227.37:4566 , piece_id=YRRPITFA3W2PA7L4KRJ6OYD5SEBCMAYVOPZA4R56D23XEEQZGHFQ, seg_pos={0 0}, upload=533798]: Uploading piece... Dec 31 00:33:43.962628120 (609h4m43.833711155s) [noise=true , address=145.79.227.2:4113 , piece_id=ZGKPJDMEGSRL6GKBFCF7CZVKCJM5PW4G6Z7LF3N7UEA3E4WMNFRA, seg_pos={0 0}, upload=533798]: Uploading piece... Dec 31 00:33:43.963204478 (609h4m43.834287513s) [noise=true , address=77.90.31.21:4877 , piece_id=S4TGVS42HN5MRIBKUR654AK2WOERFXKWPZTJZVK44UR7IN5FN55Q, seg_pos={0 0}, upload=533798]: Uploading piece... Dec 31 00:33:43.963688299 (609h4m43.834771334s) [noise=true , address=45.91.116.13:4156 , piece_id=IWYIJHXM5SF2HU337U7A3YU4ITYGZDJAKQLNKKUIZGNKFVVCQQJQ, seg_pos={0 0}, upload=533798]: Uploading piece... Dec 31 00:33:43.964191076 (609h4m43.835274111s) [noise=true , address=77.90.31.13:4627 , piece_id=UICQMYJTR5YEKUD7M4OJRED57CI2G237GT3EMYPGMWKYSCE4GRQA, seg_pos={0 0}, upload=533798]: Uploading piece... Dec 31 00:33:43.964608601 (609h4m43.835691646s) [noise=true , address=145.79.227.4:4187 , piece_id=TNOJCADMYMM2ZXMSWYWRWGJTNWVHX2CAMUMXY236S63B4MLVE43A, seg_pos={0 0}, upload=533798]: Uploading piece... Dec 31 00:33:43.965019284 (609h4m43.836102319s) [noise=true , address=145.79.227.19:4078 , piece_id=RIQ3XPKHYS25Z55CGZAMLGACVBWOJBCR5EGYOZPSCS4KZMCYNTSA, seg_pos={0 0}, upload=533798]: Uploading piece... Dec 31 00:33:43.966236792 (609h4m43.837319837s) [noise=true , address=145.79.227.50:4374 , piece_id=BLDWCE4SFJQA7ZWGDDZ5XYBBMH432M56LB7ZF3LPDB4WAVBR4GBA, seg_pos={0 0}, upload=533798]: Uploading piece... Dec 31 00:33:43.967461544 (609h4m43.838544589s) [noise=true , address=145.79.227.27:4691 , piece_id=AED64GHR5KJD75QSG3W2Q54Y4H2N3DCLK4IYWF3JLR7GXOKDOD5A, seg_pos={0 0}, upload=533798]: Uploading piece... Dec 31 00:33:43.967896762 (609h4m43.838979807s) [noise=true , address=145.79.227.36:4517 , piece_id=ZOMEYIPAPM24XKFQHAQYDZ56ZDCPAILUBC7BYS6NFHVQYKOCBCBA, seg_pos={0 0}, upload=533798]: Uploading piece... Dec 31 00:33:43.969181198 (609h4m43.840264233s) [noise=true , address=45.91.116.4:4937 , piece_id=TKRNJXBY4X2XZEPEAJAQNVHGB7EO4MWG65JXD4MMJN4AMRYNG3DA, seg_pos={0 0}, upload=533798]: Uploading piece... Dec 31 00:33:43.974310605 (609h4m43.84539366s) [noise=true , address=45.91.116.6:4976 , piece_id=BST4NLF5ZD765L5YIUXDSAXYIK2K7Z3U65JJLKGQ3KWBTCPZFRZQ, seg_pos={0 0}, upload=533798]: Uploading piece... Dec 31 00:33:43.974435924 (609h4m43.845518969s) [noise=true , address=145.79.227.14:4839 , piece_id=GY4GTCAXYTQ6IYNRZIICFQRPRTSP3HWGKITMNPMWPVGDC724Q6AQ, seg_pos={0 0}, upload=533798]: Uploading piece... Dec 31 00:33:43.975830920 (609h4m43.846913955s) [noise=true , address=45.91.116.8:4046 , piece_id=ROT7BI4CVLZTQ4OLSITC7GA64IF7QFSFFCLAN3XC4IZGYNHFMHXQ, seg_pos={0 0}, upload=533798]: Uploading piece... Dec 31 00:33:45.660804749 (609h4m45.531887784s) [upload=533798]: Next returned nil segment Dec 31 00:33:45.660820368 (609h4m45.531903413s) [upload=533798]: Waiting for error group managing segments... Dec 31 00:33:45.951050435 (609h4m45.82213347s) [noise=true , address=145.79.227.4:4187 , piece_id=TNOJCADMYMM2ZXMSWYWRWGJTNWVHX2CAMUMXY236S63B4MLVE43A, seg_pos={0 0}, upload=533798]: Done uploading piece. err: <nil> Dec 31 00:33:45.952806208 (609h4m45.823889254s) [noise=true , address=145.79.227.27:4691 , piece_id=AED64GHR5KJD75QSG3W2Q54Y4H2N3DCLK4IYWF3JLR7GXOKDOD5A, seg_pos={0 0}, upload=533798]: Done uploading piece. err: <nil> Dec 31 00:33:45.968805193 (609h4m45.839888228s) [noise=true , address=145.79.227.2:4113 , piece_id=ZGKPJDMEGSRL6GKBFCF7CZVKCJM5PW4G6Z7LF3N7UEA3E4WMNFRA, seg_pos={0 0}, upload=533798]: Done uploading piece. err: <nil> Dec 31 00:33:45.970513926 (609h4m45.841596961s) [noise=true , address=145.79.227.23:4547 , piece_id=TRELDHWOXL7AXM4POWMEWGUBBHN47JXSBQV5NUI4EEPMLIEPJLVQ, seg_pos={0 0}, upload=533798]: Done uploading piece. err: <nil> Dec 31 00:33:45.973565597 (609h4m45.844648632s) [noise=true , address=145.79.227.36:4517 , piece_id=ZOMEYIPAPM24XKFQHAQYDZ56ZDCPAILUBC7BYS6NFHVQYKOCBCBA, seg_pos={0 0}, upload=533798]: Done uploading piece. err: <nil> Dec 31 00:33:45.974458116 (609h4m45.845541152s) [noise=true , address=145.79.227.8:4329 , piece_id=CUZF4JU4IPCH6DXOEWOOHGCUTHDNAX2LOKLEXX2GFW5MAFAJ5BCA, seg_pos={0 0}, upload=533798]: Done uploading piece. err: <nil> Dec 31 00:33:45.974561093 (609h4m45.845644138s) [noise=true , address=145.79.227.37:4566 , piece_id=YRRPITFA3W2PA7L4KRJ6OYD5SEBCMAYVOPZA4R56D23XEEQZGHFQ, seg_pos={0 0}, upload=533798]: Done uploading piece. err: <nil> Dec 31 00:33:45.983584214 (609h4m45.854667259s) [noise=true , address=145.79.227.50:4374 , piece_id=BLDWCE4SFJQA7ZWGDDZ5XYBBMH432M56LB7ZF3LPDB4WAVBR4GBA, seg_pos={0 0}, upload=533798]: Done uploading piece. err: <nil> Dec 31 00:33:45.999281684 (609h4m45.870364729s) [noise=true , address=145.79.227.14:4839 , piece_id=GY4GTCAXYTQ6IYNRZIICFQRPRTSP3HWGKITMNPMWPVGDC724Q6AQ, seg_pos={0 0}, upload=533798]: Done uploading piece. err: <nil> Dec 31 00:33:46.026097925 (609h4m45.89718097s) [noise=true , address=145.79.227.19:4078 , piece_id=RIQ3XPKHYS25Z55CGZAMLGACVBWOJBCR5EGYOZPSCS4KZMCYNTSA, seg_pos={0 0}, upload=533798]: Done uploading piece. err: <nil> Dec 31 00:33:46.119472633 (609h4m45.990555678s) [noise=true , address=77.90.31.21:4877 , piece_id=S4TGVS42HN5MRIBKUR654AK2WOERFXKWPZTJZVK44UR7IN5FN55Q, seg_pos={0 0}, upload=533798]: Done uploading piece. err: <nil> Dec 31 00:33:46.223788906 (609h4m46.094871941s) [noise=true , address=77.90.31.13:4627 , piece_id=UICQMYJTR5YEKUD7M4OJRED57CI2G237GT3EMYPGMWKYSCE4GRQA, seg_pos={0 0}, upload=533798]: Done uploading piece. err: <nil> Dec 31 00:33:46.307787982 (609h4m46.178871017s) [noise=true , address=45.91.116.13:4156 , piece_id=IWYIJHXM5SF2HU337U7A3YU4ITYGZDJAKQLNKKUIZGNKFVVCQQJQ, seg_pos={0 0}, upload=533798]: Done uploading piece. err: <nil> Dec 31 00:33:46.307808380 (609h4m46.178891415s) [seg_pos={0 0}, upload=533798]: Segment reached optimal threshold of 13 pieces. Dec 31 00:33:46.308010665 (609h4m46.1790937s) [noise=true , address=45.91.116.8:4046 , piece_id=ROT7BI4CVLZTQ4OLSITC7GA64IF7QFSFFCLAN3XC4IZGYNHFMHXQ, seg_pos={0 0}, upload=533798]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H): protocol: expected piece hash; context canceled; EOF Dec 31 00:33:46.308018310 (609h4m46.179101345s) [noise=true , address=45.91.116.8:4046 , piece_id=ROT7BI4CVLZTQ4OLSITC7GA64IF7QFSFFCLAN3XC4IZGYNHFMHXQ, seg_pos={0 0}, upload=533798]: Long tail cancellation due to optimal threshold Dec 31 00:33:46.308129932 (609h4m46.179212967s) [noise=true , address=45.91.116.6:4976 , piece_id=BST4NLF5ZD765L5YIUXDSAXYIK2K7Z3U65JJLKGQ3KWBTCPZFRZQ, seg_pos={0 0}, upload=533798]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp): protocol: expected piece hash; context canceled; EOF Dec 31 00:33:46.308135213 (609h4m46.179218248s) [noise=true , address=45.91.116.6:4976 , piece_id=BST4NLF5ZD765L5YIUXDSAXYIK2K7Z3U65JJLKGQ3KWBTCPZFRZQ, seg_pos={0 0}, upload=533798]: Long tail cancellation due to optimal threshold Dec 31 00:33:46.308228550 (609h4m46.179311595s) [noise=true , address=45.91.116.4:4937 , piece_id=TKRNJXBY4X2XZEPEAJAQNVHGB7EO4MWG65JXD4MMJN4AMRYNG3DA, seg_pos={0 0}, upload=533798]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: expected piece hash; context canceled; EOF Dec 31 00:33:46.308234621 (609h4m46.179317656s) [noise=true , address=45.91.116.4:4937 , piece_id=TKRNJXBY4X2XZEPEAJAQNVHGB7EO4MWG65JXD4MMJN4AMRYNG3DA, seg_pos={0 0}, upload=533798]: Long tail cancellation due to optimal threshold Dec 31 00:33:46.308253347 (609h4m46.179336382s) [seg_pos={0 0}, upload=533798]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 00:33:46.308320215 (609h4m46.17940325s) [upload=533798]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 00:33:46.308324333 (609h4m46.179407368s) [upload=533798]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 00:33:46.594956498 (609h4m46.466039533s) [upload=533798]: Done uploading segments... Dec 31 00:33:46.594977478 (609h4m46.466060513s) [upload=533798]: Upload finished. err: <nil> Duration: 5.826754812s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [...................X...X...........X........X..X....XX.XXX..] duration: 28.147348857s trace: https://jaeger.dev.storj.io/trace/113f1a95e44d0023 FAIL (2025-12-31 00:35:55 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, Piece ID: S4TGVS42HN5MRIBKUR654AK2WOERFXKWPZTJZVK44UR7IN5FN55Q) context deadline exceeded; piecestore: (Node ID: 1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, Piece ID: UICQMYJTR5YEKUD7M4OJRED57CI2G237GT3EMYPGMWKYSCE4GRQA) context deadline exceeded ======LOGS====== Dec 31 00:33:43.715936146 (609h4m43.587019181s) [upload=533798]: Starting upload Dec 31 00:33:43.715992443 (609h4m43.587075478s) [upload=533798]: Done starting upload Dec 31 00:33:43.716132921 (609h4m43.587215976s) [upload=533798]: Uploading segments... Dec 31 00:33:43.716383137 (609h4m43.587466172s) [upload=533798]: Got next segment. Inline: false Dec 31 00:33:43.716430236 (609h4m43.587513281s) [upload=533798]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 00:33:43.716456406 (609h4m43.587539441s) [upload=533798]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 00:33:43.961837344 (609h4m43.832920389s) [seg_pos={0 0}, upload=533798]: Begin upload segment... Dec 31 00:33:43.961889664 (609h4m43.832972709s) [seg_pos={0 0}, upload=533798]: Done begin upload segment. Dec 31 00:33:43.961925643 (609h4m43.833008678s) [noise=true , address=145.79.227.23:4547 , piece_id=TRELDHWOXL7AXM4POWMEWGUBBHN47JXSBQV5NUI4EEPMLIEPJLVQ, seg_pos={0 0}, upload=533798]: Uploading piece... Dec 31 00:33:43.961935040 (609h4m43.833018075s) [noise=true , address=145.79.227.8:4329 , piece_id=CUZF4JU4IPCH6DXOEWOOHGCUTHDNAX2LOKLEXX2GFW5MAFAJ5BCA, seg_pos={0 0}, upload=533798]: Uploading piece... Dec 31 00:33:43.962101387 (609h4m43.833184422s) [noise=true , address=145.79.227.37:4566 , piece_id=YRRPITFA3W2PA7L4KRJ6OYD5SEBCMAYVOPZA4R56D23XEEQZGHFQ, seg_pos={0 0}, upload=533798]: Uploading piece... Dec 31 00:33:43.962628120 (609h4m43.833711155s) [noise=true , address=145.79.227.2:4113 , piece_id=ZGKPJDMEGSRL6GKBFCF7CZVKCJM5PW4G6Z7LF3N7UEA3E4WMNFRA, seg_pos={0 0}, upload=533798]: Uploading piece... Dec 31 00:33:43.963204478 (609h4m43.834287513s) [noise=true , address=77.90.31.21:4877 , piece_id=S4TGVS42HN5MRIBKUR654AK2WOERFXKWPZTJZVK44UR7IN5FN55Q, seg_pos={0 0}, upload=533798]: Uploading piece... Dec 31 00:33:43.963688299 (609h4m43.834771334s) [noise=true , address=45.91.116.13:4156 , piece_id=IWYIJHXM5SF2HU337U7A3YU4ITYGZDJAKQLNKKUIZGNKFVVCQQJQ, seg_pos={0 0}, upload=533798]: Uploading piece... Dec 31 00:33:43.964191076 (609h4m43.835274111s) [noise=true , address=77.90.31.13:4627 , piece_id=UICQMYJTR5YEKUD7M4OJRED57CI2G237GT3EMYPGMWKYSCE4GRQA, seg_pos={0 0}, upload=533798]: Uploading piece... Dec 31 00:33:43.964608601 (609h4m43.835691646s) [noise=true , address=145.79.227.4:4187 , piece_id=TNOJCADMYMM2ZXMSWYWRWGJTNWVHX2CAMUMXY236S63B4MLVE43A, seg_pos={0 0}, upload=533798]: Uploading piece... Dec 31 00:33:43.965019284 (609h4m43.836102319s) [noise=true , address=145.79.227.19:4078 , piece_id=RIQ3XPKHYS25Z55CGZAMLGACVBWOJBCR5EGYOZPSCS4KZMCYNTSA, seg_pos={0 0}, upload=533798]: Uploading piece... Dec 31 00:33:43.966236792 (609h4m43.837319837s) [noise=true , address=145.79.227.50:4374 , piece_id=BLDWCE4SFJQA7ZWGDDZ5XYBBMH432M56LB7ZF3LPDB4WAVBR4GBA, seg_pos={0 0}, upload=533798]: Uploading piece... Dec 31 00:33:43.967461544 (609h4m43.838544589s) [noise=true , address=145.79.227.27:4691 , piece_id=AED64GHR5KJD75QSG3W2Q54Y4H2N3DCLK4IYWF3JLR7GXOKDOD5A, seg_pos={0 0}, upload=533798]: Uploading piece... Dec 31 00:33:43.967896762 (609h4m43.838979807s) [noise=true , address=145.79.227.36:4517 , piece_id=ZOMEYIPAPM24XKFQHAQYDZ56ZDCPAILUBC7BYS6NFHVQYKOCBCBA, seg_pos={0 0}, upload=533798]: Uploading piece... Dec 31 00:33:43.969181198 (609h4m43.840264233s) [noise=true , address=45.91.116.4:4937 , piece_id=TKRNJXBY4X2XZEPEAJAQNVHGB7EO4MWG65JXD4MMJN4AMRYNG3DA, seg_pos={0 0}, upload=533798]: Uploading piece... Dec 31 00:33:43.974310605 (609h4m43.84539366s) [noise=true , address=45.91.116.6:4976 , piece_id=BST4NLF5ZD765L5YIUXDSAXYIK2K7Z3U65JJLKGQ3KWBTCPZFRZQ, seg_pos={0 0}, upload=533798]: Uploading piece... Dec 31 00:33:43.974435924 (609h4m43.845518969s) [noise=true , address=145.79.227.14:4839 , piece_id=GY4GTCAXYTQ6IYNRZIICFQRPRTSP3HWGKITMNPMWPVGDC724Q6AQ, seg_pos={0 0}, upload=533798]: Uploading piece... Dec 31 00:33:43.975830920 (609h4m43.846913955s) [noise=true , address=45.91.116.8:4046 , piece_id=ROT7BI4CVLZTQ4OLSITC7GA64IF7QFSFFCLAN3XC4IZGYNHFMHXQ, seg_pos={0 0}, upload=533798]: Uploading piece... Dec 31 00:33:45.660804749 (609h4m45.531887784s) [upload=533798]: Next returned nil segment Dec 31 00:33:45.660820368 (609h4m45.531903413s) [upload=533798]: Waiting for error group managing segments... Dec 31 00:33:45.951050435 (609h4m45.82213347s) [noise=true , address=145.79.227.4:4187 , piece_id=TNOJCADMYMM2ZXMSWYWRWGJTNWVHX2CAMUMXY236S63B4MLVE43A, seg_pos={0 0}, upload=533798]: Done uploading piece. err: <nil> Dec 31 00:33:45.952806208 (609h4m45.823889254s) [noise=true , address=145.79.227.27:4691 , piece_id=AED64GHR5KJD75QSG3W2Q54Y4H2N3DCLK4IYWF3JLR7GXOKDOD5A, seg_pos={0 0}, upload=533798]: Done uploading piece. err: <nil> Dec 31 00:33:45.968805193 (609h4m45.839888228s) [noise=true , address=145.79.227.2:4113 , piece_id=ZGKPJDMEGSRL6GKBFCF7CZVKCJM5PW4G6Z7LF3N7UEA3E4WMNFRA, seg_pos={0 0}, upload=533798]: Done uploading piece. err: <nil> Dec 31 00:33:45.970513926 (609h4m45.841596961s) [noise=true , address=145.79.227.23:4547 , piece_id=TRELDHWOXL7AXM4POWMEWGUBBHN47JXSBQV5NUI4EEPMLIEPJLVQ, seg_pos={0 0}, upload=533798]: Done uploading piece. err: <nil> Dec 31 00:33:45.973565597 (609h4m45.844648632s) [noise=true , address=145.79.227.36:4517 , piece_id=ZOMEYIPAPM24XKFQHAQYDZ56ZDCPAILUBC7BYS6NFHVQYKOCBCBA, seg_pos={0 0}, upload=533798]: Done uploading piece. err: <nil> Dec 31 00:33:45.974458116 (609h4m45.845541152s) [noise=true , address=145.79.227.8:4329 , piece_id=CUZF4JU4IPCH6DXOEWOOHGCUTHDNAX2LOKLEXX2GFW5MAFAJ5BCA, seg_pos={0 0}, upload=533798]: Done uploading piece. err: <nil> Dec 31 00:33:45.974561093 (609h4m45.845644138s) [noise=true , address=145.79.227.37:4566 , piece_id=YRRPITFA3W2PA7L4KRJ6OYD5SEBCMAYVOPZA4R56D23XEEQZGHFQ, seg_pos={0 0}, upload=533798]: Done uploading piece. err: <nil> Dec 31 00:33:45.983584214 (609h4m45.854667259s) [noise=true , address=145.79.227.50:4374 , piece_id=BLDWCE4SFJQA7ZWGDDZ5XYBBMH432M56LB7ZF3LPDB4WAVBR4GBA, seg_pos={0 0}, upload=533798]: Done uploading piece. err: <nil> Dec 31 00:33:45.999281684 (609h4m45.870364729s) [noise=true , address=145.79.227.14:4839 , piece_id=GY4GTCAXYTQ6IYNRZIICFQRPRTSP3HWGKITMNPMWPVGDC724Q6AQ, seg_pos={0 0}, upload=533798]: Done uploading piece. err: <nil> Dec 31 00:33:46.026097925 (609h4m45.89718097s) [noise=true , address=145.79.227.19:4078 , piece_id=RIQ3XPKHYS25Z55CGZAMLGACVBWOJBCR5EGYOZPSCS4KZMCYNTSA, seg_pos={0 0}, upload=533798]: Done uploading piece. err: <nil> Dec 31 00:33:46.119472633 (609h4m45.990555678s) [noise=true , address=77.90.31.21:4877 , piece_id=S4TGVS42HN5MRIBKUR654AK2WOERFXKWPZTJZVK44UR7IN5FN55Q, seg_pos={0 0}, upload=533798]: Done uploading piece. err: <nil> Dec 31 00:33:46.223788906 (609h4m46.094871941s) [noise=true , address=77.90.31.13:4627 , piece_id=UICQMYJTR5YEKUD7M4OJRED57CI2G237GT3EMYPGMWKYSCE4GRQA, seg_pos={0 0}, upload=533798]: Done uploading piece. err: <nil> Dec 31 00:33:46.307787982 (609h4m46.178871017s) [noise=true , address=45.91.116.13:4156 , piece_id=IWYIJHXM5SF2HU337U7A3YU4ITYGZDJAKQLNKKUIZGNKFVVCQQJQ, seg_pos={0 0}, upload=533798]: Done uploading piece. err: <nil> Dec 31 00:33:46.307808380 (609h4m46.178891415s) [seg_pos={0 0}, upload=533798]: Segment reached optimal threshold of 13 pieces. Dec 31 00:33:46.308010665 (609h4m46.1790937s) [noise=true , address=45.91.116.8:4046 , piece_id=ROT7BI4CVLZTQ4OLSITC7GA64IF7QFSFFCLAN3XC4IZGYNHFMHXQ, seg_pos={0 0}, upload=533798]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H): protocol: expected piece hash; context canceled; EOF Dec 31 00:33:46.308018310 (609h4m46.179101345s) [noise=true , address=45.91.116.8:4046 , piece_id=ROT7BI4CVLZTQ4OLSITC7GA64IF7QFSFFCLAN3XC4IZGYNHFMHXQ, seg_pos={0 0}, upload=533798]: Long tail cancellation due to optimal threshold Dec 31 00:33:46.308129932 (609h4m46.179212967s) [noise=true , address=45.91.116.6:4976 , piece_id=BST4NLF5ZD765L5YIUXDSAXYIK2K7Z3U65JJLKGQ3KWBTCPZFRZQ, seg_pos={0 0}, upload=533798]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp): protocol: expected piece hash; context canceled; EOF Dec 31 00:33:46.308135213 (609h4m46.179218248s) [noise=true , address=45.91.116.6:4976 , piece_id=BST4NLF5ZD765L5YIUXDSAXYIK2K7Z3U65JJLKGQ3KWBTCPZFRZQ, seg_pos={0 0}, upload=533798]: Long tail cancellation due to optimal threshold Dec 31 00:33:46.308228550 (609h4m46.179311595s) [noise=true , address=45.91.116.4:4937 , piece_id=TKRNJXBY4X2XZEPEAJAQNVHGB7EO4MWG65JXD4MMJN4AMRYNG3DA, seg_pos={0 0}, upload=533798]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: expected piece hash; context canceled; EOF Dec 31 00:33:46.308234621 (609h4m46.179317656s) [noise=true , address=45.91.116.4:4937 , piece_id=TKRNJXBY4X2XZEPEAJAQNVHGB7EO4MWG65JXD4MMJN4AMRYNG3DA, seg_pos={0 0}, upload=533798]: Long tail cancellation due to optimal threshold Dec 31 00:33:46.308253347 (609h4m46.179336382s) [seg_pos={0 0}, upload=533798]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 00:33:46.308320215 (609h4m46.17940325s) [upload=533798]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 00:33:46.308324333 (609h4m46.179407368s) [upload=533798]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 00:33:46.594956498 (609h4m46.466039533s) [upload=533798]: Done uploading segments... Dec 31 00:33:46.594977478 (609h4m46.466060513s) [upload=533798]: Upload finished. err: <nil> Duration: 28.147379194s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 28.663463427s trace: https://jaeger.dev.storj.io/trace/67f6164cdd15ab81 FAIL (2025-12-31 00:35:54 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== 31 00:33:25.026367244 (609h4m24.897450289s) [noise=true , address=145.79.227.2:4128 , piece_id=DW37Q32FFAZ37KHMOII6XGCJVDO2XTKED5TLOKG65JULWTXJIQEA, seg_pos={0 0}, upload=533792]: Uploading piece... Dec 31 00:33:25.026878247 (609h4m24.897961292s) [noise=true , address=145.79.227.44:4217 , piece_id=GW44B36EHWVCRLEM6BA5MXC3ITRNCCJATW2RXSYTNC7K7WRO2IDQ, seg_pos={0 0}, upload=533792]: Uploading piece... Dec 31 00:33:25.027303136 (609h4m24.898386171s) [noise=true , address=45.91.116.13:4161 , piece_id=K4CTNQGHL5S2GOARW2F42REGZ4QVWA2RKUNULONYX3VPVHAGONUQ, seg_pos={0 0}, upload=533792]: Uploading piece... Dec 31 00:33:25.027901605 (609h4m24.89898465s) [noise=true , address=77.90.31.13:4632 , piece_id=5KUEGWEJV5NJE2EX3S5S2G7TS5TK23ZQXV3WPA4LLHO65QJRVW6Q, seg_pos={0 0}, upload=533792]: Uploading piece... Dec 31 00:33:25.028257373 (609h4m24.899340408s) [noise=true , address=145.79.227.30:4769 , piece_id=P2ANRKDKCOTOMCZZMDIH2YRVI366UOHWO3LXV7INEDB2GW35Y52A, seg_pos={0 0}, upload=533792]: Uploading piece... Dec 31 00:33:25.028623710 (609h4m24.899706745s) [noise=true , address=145.79.227.34:4930 , piece_id=L3AVRBWPDFTZANZKO4TGITCA3WHBECKEQ2BGMLFQJYN4CR7FSADA, seg_pos={0 0}, upload=533792]: Uploading piece... Dec 31 00:33:25.028952256 (609h4m24.900035291s) [noise=true , address=145.79.227.25:4630 , piece_id=5BVJWWAW2QUQABWWZCD4EW54ZVH3K5RJNM5EXJJUGXI33O6CC24A, seg_pos={0 0}, upload=533792]: Uploading piece... Dec 31 00:33:25.029314766 (609h4m24.900397801s) [noise=true , address=145.79.227.29:4971 , piece_id=EIY6YRG5ZRMEW7IWE5X25RCTGNCU3HQ27O7Q434OAZ4EI335HU4A, seg_pos={0 0}, upload=533792]: Uploading piece... Dec 31 00:33:25.029659533 (609h4m24.900742578s) [noise=true , address=145.79.227.6:4283 , piece_id=MXSG7TVTQBR2AFONVXYWQB4DBAQEEXW226FDC6SSVG5TTWGPAWFQ, seg_pos={0 0}, upload=533792]: Uploading piece... Dec 31 00:33:25.030576118 (609h4m24.901659153s) [noise=true , address=77.90.31.18:4835 , piece_id=GRMJTN6ZDDYITHJWX5UGIAGLCOW6AKADHTK6UCVGRB437TUTDFGA, seg_pos={0 0}, upload=533792]: Uploading piece... Dec 31 00:33:25.031413452 (609h4m24.902496487s) [noise=true , address=64.49.13.155:4913 , piece_id=E4WJCMYJVURPHENWEWR46KOAUAOZU7FW7WLC53DAU44BE7R3CHMQ, seg_pos={0 0}, upload=533792]: Uploading piece... Dec 31 00:33:25.031987204 (609h4m24.903070239s) [noise=true , address=145.79.227.49:4370 , piece_id=52YCNJY7BLJ3N3HDWBZCM7PP2ZPHHBS4GV4QS4WM2AWSLYKB5BBA, seg_pos={0 0}, upload=533792]: Uploading piece... Dec 31 00:33:25.032849947 (609h4m24.903932993s) [noise=true , address=145.79.227.4:4189 , piece_id=ZZF4DSIPQBWKBD45UR2J5P75U5VVMTVB54NEX5QKZV4K2WJZ43BQ, seg_pos={0 0}, upload=533792]: Uploading piece... Dec 31 00:33:25.034222320 (609h4m24.905305355s) [noise=true , address=145.79.227.49:4353 , piece_id=MVTAW6QPJY2LRNEGISHNDP7AC5ZIIUR5N5BBZRFO5I6VYJU7UR3Q, seg_pos={0 0}, upload=533792]: Uploading piece... Dec 31 00:33:25.035359376 (609h4m24.906442411s) [noise=true , address=145.79.227.35:4494 , piece_id=EY2OJLQIF4CH7DQWJWXSBJMQI37NAXSBLQP4ZPMU3IBRVLWJVYWA, seg_pos={0 0}, upload=533792]: Uploading piece... Dec 31 00:33:25.035852344 (609h4m24.906935379s) [noise=true , address=145.79.227.33:4873 , piece_id=2DSEGSC5UWL76NEXSE7WXOGSQNRNXALAINY52TLQHB57LTS2R6ZQ, seg_pos={0 0}, upload=533792]: Uploading piece... Dec 31 00:33:26.491429069 (609h4m26.362512104s) [upload=533792]: Next returned nil segment Dec 31 00:33:26.491441904 (609h4m26.362524939s) [upload=533792]: Waiting for error group managing segments... Dec 31 00:33:26.876786140 (609h4m26.747869175s) [noise=true , address=145.79.227.6:4283 , piece_id=MXSG7TVTQBR2AFONVXYWQB4DBAQEEXW226FDC6SSVG5TTWGPAWFQ, seg_pos={0 0}, upload=533792]: Done uploading piece. err: <nil> Dec 31 00:33:26.895923271 (609h4m26.767006316s) [noise=true , address=145.79.227.35:4494 , piece_id=EY2OJLQIF4CH7DQWJWXSBJMQI37NAXSBLQP4ZPMU3IBRVLWJVYWA, seg_pos={0 0}, upload=533792]: Done uploading piece. err: <nil> Dec 31 00:33:26.903106888 (609h4m26.774189923s) [noise=true , address=145.79.227.33:4873 , piece_id=2DSEGSC5UWL76NEXSE7WXOGSQNRNXALAINY52TLQHB57LTS2R6ZQ, seg_pos={0 0}, upload=533792]: Done uploading piece. err: <nil> Dec 31 00:33:26.912484285 (609h4m26.78356733s) [noise=true , address=145.79.227.29:4971 , piece_id=EIY6YRG5ZRMEW7IWE5X25RCTGNCU3HQ27O7Q434OAZ4EI335HU4A, seg_pos={0 0}, upload=533792]: Done uploading piece. err: <nil> Dec 31 00:33:27.004280888 (609h4m26.875363933s) [noise=true , address=145.79.227.21:4495 , piece_id=YWETKWGBGTY3LALK4DXWSKUTM2TFEYB4N5A3KNKW346V3F5XRN2Q, seg_pos={0 0}, upload=533792]: Done uploading piece. err: <nil> Dec 31 00:33:27.118759862 (609h4m26.989842897s) [noise=true , address=77.90.31.13:4632 , piece_id=5KUEGWEJV5NJE2EX3S5S2G7TS5TK23ZQXV3WPA4LLHO65QJRVW6Q, seg_pos={0 0}, upload=533792]: Done uploading piece. err: <nil> Dec 31 00:33:27.225324635 (609h4m27.09640767s) [noise=true , address=45.91.116.13:4161 , piece_id=K4CTNQGHL5S2GOARW2F42REGZ4QVWA2RKUNULONYX3VPVHAGONUQ, seg_pos={0 0}, upload=533792]: Done uploading piece. err: <nil> Dec 31 00:33:27.347765488 (609h4m27.218848533s) [noise=true , address=45.91.116.12:4248 , piece_id=QXKIUKBDX2GSJ2EQPALFCLPPLNWBPL3GRAEHLPJNSWUDIMU7WHYA, seg_pos={0 0}, upload=533792]: Done uploading piece. err: <nil> Dec 31 00:33:27.789508217 (609h4m27.660591252s) [noise=true , address=145.79.227.44:4217 , piece_id=GW44B36EHWVCRLEM6BA5MXC3ITRNCCJATW2RXSYTNC7K7WRO2IDQ, seg_pos={0 0}, upload=533792]: Done uploading piece. err: <nil> Dec 31 00:33:27.821042021 (609h4m27.692125056s) [noise=true , address=145.79.227.49:4353 , piece_id=MVTAW6QPJY2LRNEGISHNDP7AC5ZIIUR5N5BBZRFO5I6VYJU7UR3Q, seg_pos={0 0}, upload=533792]: Done uploading piece. err: <nil> Dec 31 00:33:27.828648224 (609h4m27.699731259s) [noise=true , address=145.79.227.49:4370 , piece_id=52YCNJY7BLJ3N3HDWBZCM7PP2ZPHHBS4GV4QS4WM2AWSLYKB5BBA, seg_pos={0 0}, upload=533792]: Done uploading piece. err: <nil> Dec 31 00:33:27.840271497 (609h4m27.711354542s) [noise=true , address=145.79.227.30:4769 , piece_id=P2ANRKDKCOTOMCZZMDIH2YRVI366UOHWO3LXV7INEDB2GW35Y52A, seg_pos={0 0}, upload=533792]: Done uploading piece. err: <nil> Dec 31 00:33:27.844064569 (609h4m27.715147614s) [noise=true , address=145.79.227.34:4930 , piece_id=L3AVRBWPDFTZANZKO4TGITCA3WHBECKEQ2BGMLFQJYN4CR7FSADA, seg_pos={0 0}, upload=533792]: Done uploading piece. err: <nil> Dec 31 00:33:27.867248213 (609h4m27.738331248s) [noise=true , address=77.90.31.18:4835 , piece_id=GRMJTN6ZDDYITHJWX5UGIAGLCOW6AKADHTK6UCVGRB437TUTDFGA, seg_pos={0 0}, upload=533792]: Done uploading piece. err: <nil> Dec 31 00:33:27.867270586 (609h4m27.738353621s) [seg_pos={0 0}, upload=533792]: Segment reached optimal threshold of 14 pieces. Dec 31 00:33:27.867476859 (609h4m27.738559904s) [noise=true , address=64.49.13.155:4913 , piece_id=E4WJCMYJVURPHENWEWR46KOAUAOZU7FW7WLC53DAU44BE7R3CHMQ, seg_pos={0 0}, upload=533792]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oyYUnWDyFo1npaKpBxMG5K31oUzJnTpcU1XHFmKux3MCVUPZV): protocol: expected piece hash; context canceled; EOF Dec 31 00:33:27.867485555 (609h4m27.73856859s) [noise=true , address=64.49.13.155:4913 , piece_id=E4WJCMYJVURPHENWEWR46KOAUAOZU7FW7WLC53DAU44BE7R3CHMQ, seg_pos={0 0}, upload=533792]: Long tail cancellation due to optimal threshold Dec 31 00:33:27.867516204 (609h4m27.738599249s) [noise=true , address=145.79.227.25:4630 , piece_id=5BVJWWAW2QUQABWWZCD4EW54ZVH3K5RJNM5EXJJUGXI33O6CC24A, seg_pos={0 0}, upload=533792]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12aETKgTzKw8fyYqbeDA2JzfxZnWn12xUPYK1eRt6TvBPPdRJDp): protocol: expected piece hash; context canceled; EOF Dec 31 00:33:27.867526553 (609h4m27.738609588s) [noise=true , address=145.79.227.25:4630 , piece_id=5BVJWWAW2QUQABWWZCD4EW54ZVH3K5RJNM5EXJJUGXI33O6CC24A, seg_pos={0 0}, upload=533792]: Long tail cancellation due to optimal threshold Dec 31 00:33:27.867714432 (609h4m27.738797477s) [noise=true , address=145.79.227.4:4189 , piece_id=ZZF4DSIPQBWKBD45UR2J5P75U5VVMTVB54NEX5QKZV4K2WJZ43BQ, seg_pos={0 0}, upload=533792]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jH3757i2F3q18XSpoXznsysZPaXmEnqynpLPDupgRfxJrTgKv): protocol: write tcp 188.166.225.170:57528->145.79.227.4:4189: use of closed network connection; write tcp 188.166.225.170:57528->145.79.227.4:4189: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57528->145.79.227.4:4189: use of closed network connection Dec 31 00:33:27.867723689 (609h4m27.738806724s) [noise=true , address=145.79.227.4:4189 , piece_id=ZZF4DSIPQBWKBD45UR2J5P75U5VVMTVB54NEX5QKZV4K2WJZ43BQ, seg_pos={0 0}, upload=533792]: Long tail cancellation due to optimal threshold Dec 31 00:33:27.867744699 (609h4m27.738827734s) [noise=true , address=145.79.227.2:4128 , piece_id=DW37Q32FFAZ37KHMOII6XGCJVDO2XTKED5TLOKG65JULWTXJIQEA, seg_pos={0 0}, upload=533792]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1wABkAtAjoYkKyNoAiaVCxSvY6WgvHvAYFPCjpNojtpiYiGtDS): protocol: expected piece hash; context canceled; EOF Dec 31 00:33:27.867753616 (609h4m27.738836651s) [noise=true , address=145.79.227.2:4128 , piece_id=DW37Q32FFAZ37KHMOII6XGCJVDO2XTKED5TLOKG65JULWTXJIQEA, seg_pos={0 0}, upload=533792]: Long tail cancellation due to optimal threshold Dec 31 00:33:27.867777462 (609h4m27.738860507s) [seg_pos={0 0}, upload=533792]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 31 00:33:27.867853687 (609h4m27.738936722s) [upload=533792]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 00:33:27.867859989 (609h4m27.738943024s) [upload=533792]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 00:33:28.167335511 (609h4m28.038418546s) [upload=533792]: Done uploading segments... Dec 31 00:33:28.167355899 (609h4m28.038438934s) [upload=533792]: Upload finished. err: <nil> Duration: 28.663490149s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 40.069943716s trace: https://jaeger.dev.storj.io/trace/67f6164cdd15ab81 FAIL (2025-12-31 00:35:04 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== 31 00:33:25.026367244 (609h4m24.897450289s) [noise=true , address=145.79.227.2:4128 , piece_id=DW37Q32FFAZ37KHMOII6XGCJVDO2XTKED5TLOKG65JULWTXJIQEA, seg_pos={0 0}, upload=533792]: Uploading piece... Dec 31 00:33:25.026878247 (609h4m24.897961292s) [noise=true , address=145.79.227.44:4217 , piece_id=GW44B36EHWVCRLEM6BA5MXC3ITRNCCJATW2RXSYTNC7K7WRO2IDQ, seg_pos={0 0}, upload=533792]: Uploading piece... Dec 31 00:33:25.027303136 (609h4m24.898386171s) [noise=true , address=45.91.116.13:4161 , piece_id=K4CTNQGHL5S2GOARW2F42REGZ4QVWA2RKUNULONYX3VPVHAGONUQ, seg_pos={0 0}, upload=533792]: Uploading piece... Dec 31 00:33:25.027901605 (609h4m24.89898465s) [noise=true , address=77.90.31.13:4632 , piece_id=5KUEGWEJV5NJE2EX3S5S2G7TS5TK23ZQXV3WPA4LLHO65QJRVW6Q, seg_pos={0 0}, upload=533792]: Uploading piece... Dec 31 00:33:25.028257373 (609h4m24.899340408s) [noise=true , address=145.79.227.30:4769 , piece_id=P2ANRKDKCOTOMCZZMDIH2YRVI366UOHWO3LXV7INEDB2GW35Y52A, seg_pos={0 0}, upload=533792]: Uploading piece... Dec 31 00:33:25.028623710 (609h4m24.899706745s) [noise=true , address=145.79.227.34:4930 , piece_id=L3AVRBWPDFTZANZKO4TGITCA3WHBECKEQ2BGMLFQJYN4CR7FSADA, seg_pos={0 0}, upload=533792]: Uploading piece... Dec 31 00:33:25.028952256 (609h4m24.900035291s) [noise=true , address=145.79.227.25:4630 , piece_id=5BVJWWAW2QUQABWWZCD4EW54ZVH3K5RJNM5EXJJUGXI33O6CC24A, seg_pos={0 0}, upload=533792]: Uploading piece... Dec 31 00:33:25.029314766 (609h4m24.900397801s) [noise=true , address=145.79.227.29:4971 , piece_id=EIY6YRG5ZRMEW7IWE5X25RCTGNCU3HQ27O7Q434OAZ4EI335HU4A, seg_pos={0 0}, upload=533792]: Uploading piece... Dec 31 00:33:25.029659533 (609h4m24.900742578s) [noise=true , address=145.79.227.6:4283 , piece_id=MXSG7TVTQBR2AFONVXYWQB4DBAQEEXW226FDC6SSVG5TTWGPAWFQ, seg_pos={0 0}, upload=533792]: Uploading piece... Dec 31 00:33:25.030576118 (609h4m24.901659153s) [noise=true , address=77.90.31.18:4835 , piece_id=GRMJTN6ZDDYITHJWX5UGIAGLCOW6AKADHTK6UCVGRB437TUTDFGA, seg_pos={0 0}, upload=533792]: Uploading piece... Dec 31 00:33:25.031413452 (609h4m24.902496487s) [noise=true , address=64.49.13.155:4913 , piece_id=E4WJCMYJVURPHENWEWR46KOAUAOZU7FW7WLC53DAU44BE7R3CHMQ, seg_pos={0 0}, upload=533792]: Uploading piece... Dec 31 00:33:25.031987204 (609h4m24.903070239s) [noise=true , address=145.79.227.49:4370 , piece_id=52YCNJY7BLJ3N3HDWBZCM7PP2ZPHHBS4GV4QS4WM2AWSLYKB5BBA, seg_pos={0 0}, upload=533792]: Uploading piece... Dec 31 00:33:25.032849947 (609h4m24.903932993s) [noise=true , address=145.79.227.4:4189 , piece_id=ZZF4DSIPQBWKBD45UR2J5P75U5VVMTVB54NEX5QKZV4K2WJZ43BQ, seg_pos={0 0}, upload=533792]: Uploading piece... Dec 31 00:33:25.034222320 (609h4m24.905305355s) [noise=true , address=145.79.227.49:4353 , piece_id=MVTAW6QPJY2LRNEGISHNDP7AC5ZIIUR5N5BBZRFO5I6VYJU7UR3Q, seg_pos={0 0}, upload=533792]: Uploading piece... Dec 31 00:33:25.035359376 (609h4m24.906442411s) [noise=true , address=145.79.227.35:4494 , piece_id=EY2OJLQIF4CH7DQWJWXSBJMQI37NAXSBLQP4ZPMU3IBRVLWJVYWA, seg_pos={0 0}, upload=533792]: Uploading piece... Dec 31 00:33:25.035852344 (609h4m24.906935379s) [noise=true , address=145.79.227.33:4873 , piece_id=2DSEGSC5UWL76NEXSE7WXOGSQNRNXALAINY52TLQHB57LTS2R6ZQ, seg_pos={0 0}, upload=533792]: Uploading piece... Dec 31 00:33:26.491429069 (609h4m26.362512104s) [upload=533792]: Next returned nil segment Dec 31 00:33:26.491441904 (609h4m26.362524939s) [upload=533792]: Waiting for error group managing segments... Dec 31 00:33:26.876786140 (609h4m26.747869175s) [noise=true , address=145.79.227.6:4283 , piece_id=MXSG7TVTQBR2AFONVXYWQB4DBAQEEXW226FDC6SSVG5TTWGPAWFQ, seg_pos={0 0}, upload=533792]: Done uploading piece. err: <nil> Dec 31 00:33:26.895923271 (609h4m26.767006316s) [noise=true , address=145.79.227.35:4494 , piece_id=EY2OJLQIF4CH7DQWJWXSBJMQI37NAXSBLQP4ZPMU3IBRVLWJVYWA, seg_pos={0 0}, upload=533792]: Done uploading piece. err: <nil> Dec 31 00:33:26.903106888 (609h4m26.774189923s) [noise=true , address=145.79.227.33:4873 , piece_id=2DSEGSC5UWL76NEXSE7WXOGSQNRNXALAINY52TLQHB57LTS2R6ZQ, seg_pos={0 0}, upload=533792]: Done uploading piece. err: <nil> Dec 31 00:33:26.912484285 (609h4m26.78356733s) [noise=true , address=145.79.227.29:4971 , piece_id=EIY6YRG5ZRMEW7IWE5X25RCTGNCU3HQ27O7Q434OAZ4EI335HU4A, seg_pos={0 0}, upload=533792]: Done uploading piece. err: <nil> Dec 31 00:33:27.004280888 (609h4m26.875363933s) [noise=true , address=145.79.227.21:4495 , piece_id=YWETKWGBGTY3LALK4DXWSKUTM2TFEYB4N5A3KNKW346V3F5XRN2Q, seg_pos={0 0}, upload=533792]: Done uploading piece. err: <nil> Dec 31 00:33:27.118759862 (609h4m26.989842897s) [noise=true , address=77.90.31.13:4632 , piece_id=5KUEGWEJV5NJE2EX3S5S2G7TS5TK23ZQXV3WPA4LLHO65QJRVW6Q, seg_pos={0 0}, upload=533792]: Done uploading piece. err: <nil> Dec 31 00:33:27.225324635 (609h4m27.09640767s) [noise=true , address=45.91.116.13:4161 , piece_id=K4CTNQGHL5S2GOARW2F42REGZ4QVWA2RKUNULONYX3VPVHAGONUQ, seg_pos={0 0}, upload=533792]: Done uploading piece. err: <nil> Dec 31 00:33:27.347765488 (609h4m27.218848533s) [noise=true , address=45.91.116.12:4248 , piece_id=QXKIUKBDX2GSJ2EQPALFCLPPLNWBPL3GRAEHLPJNSWUDIMU7WHYA, seg_pos={0 0}, upload=533792]: Done uploading piece. err: <nil> Dec 31 00:33:27.789508217 (609h4m27.660591252s) [noise=true , address=145.79.227.44:4217 , piece_id=GW44B36EHWVCRLEM6BA5MXC3ITRNCCJATW2RXSYTNC7K7WRO2IDQ, seg_pos={0 0}, upload=533792]: Done uploading piece. err: <nil> Dec 31 00:33:27.821042021 (609h4m27.692125056s) [noise=true , address=145.79.227.49:4353 , piece_id=MVTAW6QPJY2LRNEGISHNDP7AC5ZIIUR5N5BBZRFO5I6VYJU7UR3Q, seg_pos={0 0}, upload=533792]: Done uploading piece. err: <nil> Dec 31 00:33:27.828648224 (609h4m27.699731259s) [noise=true , address=145.79.227.49:4370 , piece_id=52YCNJY7BLJ3N3HDWBZCM7PP2ZPHHBS4GV4QS4WM2AWSLYKB5BBA, seg_pos={0 0}, upload=533792]: Done uploading piece. err: <nil> Dec 31 00:33:27.840271497 (609h4m27.711354542s) [noise=true , address=145.79.227.30:4769 , piece_id=P2ANRKDKCOTOMCZZMDIH2YRVI366UOHWO3LXV7INEDB2GW35Y52A, seg_pos={0 0}, upload=533792]: Done uploading piece. err: <nil> Dec 31 00:33:27.844064569 (609h4m27.715147614s) [noise=true , address=145.79.227.34:4930 , piece_id=L3AVRBWPDFTZANZKO4TGITCA3WHBECKEQ2BGMLFQJYN4CR7FSADA, seg_pos={0 0}, upload=533792]: Done uploading piece. err: <nil> Dec 31 00:33:27.867248213 (609h4m27.738331248s) [noise=true , address=77.90.31.18:4835 , piece_id=GRMJTN6ZDDYITHJWX5UGIAGLCOW6AKADHTK6UCVGRB437TUTDFGA, seg_pos={0 0}, upload=533792]: Done uploading piece. err: <nil> Dec 31 00:33:27.867270586 (609h4m27.738353621s) [seg_pos={0 0}, upload=533792]: Segment reached optimal threshold of 14 pieces. Dec 31 00:33:27.867476859 (609h4m27.738559904s) [noise=true , address=64.49.13.155:4913 , piece_id=E4WJCMYJVURPHENWEWR46KOAUAOZU7FW7WLC53DAU44BE7R3CHMQ, seg_pos={0 0}, upload=533792]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oyYUnWDyFo1npaKpBxMG5K31oUzJnTpcU1XHFmKux3MCVUPZV): protocol: expected piece hash; context canceled; EOF Dec 31 00:33:27.867485555 (609h4m27.73856859s) [noise=true , address=64.49.13.155:4913 , piece_id=E4WJCMYJVURPHENWEWR46KOAUAOZU7FW7WLC53DAU44BE7R3CHMQ, seg_pos={0 0}, upload=533792]: Long tail cancellation due to optimal threshold Dec 31 00:33:27.867516204 (609h4m27.738599249s) [noise=true , address=145.79.227.25:4630 , piece_id=5BVJWWAW2QUQABWWZCD4EW54ZVH3K5RJNM5EXJJUGXI33O6CC24A, seg_pos={0 0}, upload=533792]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12aETKgTzKw8fyYqbeDA2JzfxZnWn12xUPYK1eRt6TvBPPdRJDp): protocol: expected piece hash; context canceled; EOF Dec 31 00:33:27.867526553 (609h4m27.738609588s) [noise=true , address=145.79.227.25:4630 , piece_id=5BVJWWAW2QUQABWWZCD4EW54ZVH3K5RJNM5EXJJUGXI33O6CC24A, seg_pos={0 0}, upload=533792]: Long tail cancellation due to optimal threshold Dec 31 00:33:27.867714432 (609h4m27.738797477s) [noise=true , address=145.79.227.4:4189 , piece_id=ZZF4DSIPQBWKBD45UR2J5P75U5VVMTVB54NEX5QKZV4K2WJZ43BQ, seg_pos={0 0}, upload=533792]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jH3757i2F3q18XSpoXznsysZPaXmEnqynpLPDupgRfxJrTgKv): protocol: write tcp 188.166.225.170:57528->145.79.227.4:4189: use of closed network connection; write tcp 188.166.225.170:57528->145.79.227.4:4189: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57528->145.79.227.4:4189: use of closed network connection Dec 31 00:33:27.867723689 (609h4m27.738806724s) [noise=true , address=145.79.227.4:4189 , piece_id=ZZF4DSIPQBWKBD45UR2J5P75U5VVMTVB54NEX5QKZV4K2WJZ43BQ, seg_pos={0 0}, upload=533792]: Long tail cancellation due to optimal threshold Dec 31 00:33:27.867744699 (609h4m27.738827734s) [noise=true , address=145.79.227.2:4128 , piece_id=DW37Q32FFAZ37KHMOII6XGCJVDO2XTKED5TLOKG65JULWTXJIQEA, seg_pos={0 0}, upload=533792]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1wABkAtAjoYkKyNoAiaVCxSvY6WgvHvAYFPCjpNojtpiYiGtDS): protocol: expected piece hash; context canceled; EOF Dec 31 00:33:27.867753616 (609h4m27.738836651s) [noise=true , address=145.79.227.2:4128 , piece_id=DW37Q32FFAZ37KHMOII6XGCJVDO2XTKED5TLOKG65JULWTXJIQEA, seg_pos={0 0}, upload=533792]: Long tail cancellation due to optimal threshold Dec 31 00:33:27.867777462 (609h4m27.738860507s) [seg_pos={0 0}, upload=533792]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 31 00:33:27.867853687 (609h4m27.738936722s) [upload=533792]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 00:33:27.867859989 (609h4m27.738943024s) [upload=533792]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 00:33:28.167335511 (609h4m28.038418546s) [upload=533792]: Done uploading segments... Dec 31 00:33:28.167355899 (609h4m28.038438934s) [upload=533792]: Upload finished. err: <nil> Duration: 40.069969565s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)