Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [X....X...X.X........X.......X..X.XXXXXXX...X.........XXXX..X] duration: 30.002149484s trace: https://jaeger.dev.storj.io/trace/7c75d88dbfcfe426 FAIL (2025-12-27 18:35:37 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, Piece ID: GUPMFUQBHSKHVKR3J4Y7EZCBEHRMQ3UDACW6T73XVDSZYFAQJ3OA) context deadline exceeded; piecestore: (Node ID: 1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, Piece ID: PUA63GSD2C7NWJQOZQIIZ7F37NGC5J2E57UK4XVBJASDPQCIOQZA) context deadline exceeded ======LOGS====== 27 18:35:08.238804493 (531h6m8.109887538s) [upload=465774]: Uploading segments... Dec 27 18:35:08.238970550 (531h6m8.110053585s) [upload=465774]: Got next segment. Inline: false Dec 27 18:35:08.239018821 (531h6m8.110101866s) [upload=465774]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 18:35:08.239023992 (531h6m8.110107037s) [upload=465774]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 18:35:08.486211550 (531h6m8.357294595s) [seg_pos={0 0}, upload=465774]: Begin upload segment... Dec 27 18:35:08.486269370 (531h6m8.357352405s) [seg_pos={0 0}, upload=465774]: Done begin upload segment. Dec 27 18:35:08.486321118 (531h6m8.357404163s) [noise=true , address=145.79.227.24:4581 , piece_id=5ABRXLJKI5SBBB3HAJI5ER3AJFJ5NHZVQAJ4XUEI2G2MIY3SO3YA, seg_pos={0 0}, upload=465774]: Uploading piece... Dec 27 18:35:08.486388587 (531h6m8.357471632s) [noise=true , address=145.79.227.25:4618 , piece_id=QEKNJSH252DQKNBLPAEHC7HKPVRS7UKW3EK62U4UAU6F2FOTAE7Q, seg_pos={0 0}, upload=465774]: Uploading piece... Dec 27 18:35:08.487122975 (531h6m8.35820601s) [noise=true , address=145.79.227.17:4972 , piece_id=SARLRYE7USNNHQIYHM3RBKZTVSGVVGWNEYNVVJ4GRL2Y4XD4ABMA, seg_pos={0 0}, upload=465774]: Uploading piece... Dec 27 18:35:08.487582119 (531h6m8.358665164s) [noise=true , address=145.79.227.38:4979 , piece_id=Y7F2ZGWVIWIARLRVVMIINE3YPRCN2EOWVUBEOS7XP7HPKF4WGV3Q, seg_pos={0 0}, upload=465774]: Uploading piece... Dec 27 18:35:08.488018531 (531h6m8.359101576s) [noise=true , address=45.91.116.4:4940 , piece_id=5KUW3RSQGNU5N6PSQ6MFLPZAZHKAOVEZ7KBC5EEWNN2CZJIR4E7Q, seg_pos={0 0}, upload=465774]: Uploading piece... Dec 27 18:35:08.488459420 (531h6m8.359542465s) [noise=true , address=145.79.227.15:4874 , piece_id=YNHB4THKJ3USOYQGGTEDK7XF5PNZ6WGUBPX4KLWLXQHBTAHOINIQ, seg_pos={0 0}, upload=465774]: Uploading piece... Dec 27 18:35:08.488904497 (531h6m8.359987542s) [noise=true , address=145.79.227.34:4906 , piece_id=K3FDGUMH44QTYEQJ7CKHQZ7VLQWA3CLPZ47QU4PUL263TZV3U53A, seg_pos={0 0}, upload=465774]: Uploading piece... Dec 27 18:35:08.489368151 (531h6m8.360451196s) [noise=true , address=45.91.116.14:4192 , piece_id=DHH5SDL4KVBRV2CLY5CHMKPRQFFVQNWIGS6C5R3NEXTANU7YVHVA, seg_pos={0 0}, upload=465774]: Uploading piece... Dec 27 18:35:08.490777654 (531h6m8.361860699s) [noise=true , address=145.79.227.18:4007 , piece_id=UU4PBZ4MOHVJSXZHJ5YK6EQ43LG2CMTLLPVBTSEL6GCAYGCZBJUQ, seg_pos={0 0}, upload=465774]: Uploading piece... Dec 27 18:35:08.491245285 (531h6m8.36232832s) [noise=true , address=145.79.227.47:4266 , piece_id=FO2DL6Y6JRCN3IYCC32SWMC32FXSIC6VNH4HCQJA7P25UJMKIKLQ, seg_pos={0 0}, upload=465774]: Uploading piece... Dec 27 18:35:08.492239959 (531h6m8.363323004s) [noise=true , address=145.79.227.11:4732 , piece_id=YS7OPOPFVPHW2PKYJ3XFLBXZ2NPGLENB3UDBWZLHJSTH7W5IN4OQ, seg_pos={0 0}, upload=465774]: Uploading piece... Dec 27 18:35:08.493633743 (531h6m8.364716788s) [noise=true , address=77.90.31.10:4519 , piece_id=GUPMFUQBHSKHVKR3J4Y7EZCBEHRMQ3UDACW6T73XVDSZYFAQJ3OA, seg_pos={0 0}, upload=465774]: Uploading piece... Dec 27 18:35:08.495030572 (531h6m8.366113617s) [noise=true , address=45.91.116.7:4010 , piece_id=PUA63GSD2C7NWJQOZQIIZ7F37NGC5J2E57UK4XVBJASDPQCIOQZA, seg_pos={0 0}, upload=465774]: Uploading piece... Dec 27 18:35:08.496798679 (531h6m8.367881724s) [noise=true , address=145.79.227.12:4765 , piece_id=VISMRBWDJ2BBCJGDWB6IMKPGAIBSBRFTSFOMHGJO27LCPNYGZGYA, seg_pos={0 0}, upload=465774]: Uploading piece... Dec 27 18:35:08.500940837 (531h6m8.372023882s) [noise=true , address=145.79.227.9:4365 , piece_id=4NUDWZ4B4GKGO7WVB5VKHPOLGWGNEP6WFB3NBHHJ27OSIT5QF64A, seg_pos={0 0}, upload=465774]: Uploading piece... Dec 27 18:35:08.505121487 (531h6m8.376204532s) [noise=true , address=77.90.31.14:4664 , piece_id=5QUNP6U67WATL7SK5DPAPUOMKMTE2M5ZFPS4VAFUQ3ZBPQWXW2SQ, seg_pos={0 0}, upload=465774]: Uploading piece... Dec 27 18:35:10.110566599 (531h6m9.981649644s) [upload=465774]: Next returned nil segment Dec 27 18:35:10.110585285 (531h6m9.98166833s) [upload=465774]: Waiting for error group managing segments... Dec 27 18:35:10.495280874 (531h6m10.366363919s) [noise=true , address=145.79.227.17:4972 , piece_id=SARLRYE7USNNHQIYHM3RBKZTVSGVVGWNEYNVVJ4GRL2Y4XD4ABMA, seg_pos={0 0}, upload=465774]: Done uploading piece. err: <nil> Dec 27 18:35:10.495851189 (531h6m10.366934224s) [noise=true , address=145.79.227.18:4007 , piece_id=UU4PBZ4MOHVJSXZHJ5YK6EQ43LG2CMTLLPVBTSEL6GCAYGCZBJUQ, seg_pos={0 0}, upload=465774]: Done uploading piece. err: <nil> Dec 27 18:35:10.499044580 (531h6m10.370127615s) [noise=true , address=145.79.227.11:4732 , piece_id=YS7OPOPFVPHW2PKYJ3XFLBXZ2NPGLENB3UDBWZLHJSTH7W5IN4OQ, seg_pos={0 0}, upload=465774]: Done uploading piece. err: <nil> Dec 27 18:35:10.504516910 (531h6m10.375599955s) [noise=true , address=145.79.227.38:4979 , piece_id=Y7F2ZGWVIWIARLRVVMIINE3YPRCN2EOWVUBEOS7XP7HPKF4WGV3Q, seg_pos={0 0}, upload=465774]: Done uploading piece. err: <nil> Dec 27 18:35:10.512907297 (531h6m10.383990332s) [noise=true , address=145.79.227.12:4765 , piece_id=VISMRBWDJ2BBCJGDWB6IMKPGAIBSBRFTSFOMHGJO27LCPNYGZGYA, seg_pos={0 0}, upload=465774]: Done uploading piece. err: <nil> Dec 27 18:35:10.513607580 (531h6m10.384690625s) [noise=true , address=145.79.227.47:4266 , piece_id=FO2DL6Y6JRCN3IYCC32SWMC32FXSIC6VNH4HCQJA7P25UJMKIKLQ, seg_pos={0 0}, upload=465774]: Done uploading piece. err: <nil> Dec 27 18:35:10.515158554 (531h6m10.386241599s) [noise=true , address=145.79.227.24:4581 , piece_id=5ABRXLJKI5SBBB3HAJI5ER3AJFJ5NHZVQAJ4XUEI2G2MIY3SO3YA, seg_pos={0 0}, upload=465774]: Done uploading piece. err: <nil> Dec 27 18:35:10.515748057 (531h6m10.386831102s) [noise=true , address=145.79.227.9:4365 , piece_id=4NUDWZ4B4GKGO7WVB5VKHPOLGWGNEP6WFB3NBHHJ27OSIT5QF64A, seg_pos={0 0}, upload=465774]: Done uploading piece. err: <nil> Dec 27 18:35:10.516236036 (531h6m10.387319071s) [noise=true , address=145.79.227.25:4618 , piece_id=QEKNJSH252DQKNBLPAEHC7HKPVRS7UKW3EK62U4UAU6F2FOTAE7Q, seg_pos={0 0}, upload=465774]: Done uploading piece. err: <nil> Dec 27 18:35:10.519369963 (531h6m10.390452998s) [noise=true , address=145.79.227.15:4874 , piece_id=YNHB4THKJ3USOYQGGTEDK7XF5PNZ6WGUBPX4KLWLXQHBTAHOINIQ, seg_pos={0 0}, upload=465774]: Done uploading piece. err: <nil> Dec 27 18:35:10.664436865 (531h6m10.53551991s) [noise=true , address=77.90.31.10:4519 , piece_id=GUPMFUQBHSKHVKR3J4Y7EZCBEHRMQ3UDACW6T73XVDSZYFAQJ3OA, seg_pos={0 0}, upload=465774]: Done uploading piece. err: <nil> Dec 27 18:35:10.715523444 (531h6m10.586606489s) [noise=true , address=45.91.116.14:4192 , piece_id=DHH5SDL4KVBRV2CLY5CHMKPRQFFVQNWIGS6C5R3NEXTANU7YVHVA, seg_pos={0 0}, upload=465774]: Done uploading piece. err: <nil> Dec 27 18:35:10.796093974 (531h6m10.667177019s) [noise=true , address=45.91.116.7:4010 , piece_id=PUA63GSD2C7NWJQOZQIIZ7F37NGC5J2E57UK4XVBJASDPQCIOQZA, seg_pos={0 0}, upload=465774]: Done uploading piece. err: <nil> Dec 27 18:35:10.796123150 (531h6m10.667206185s) [seg_pos={0 0}, upload=465774]: Segment reached optimal threshold of 13 pieces. Dec 27 18:35:10.796344090 (531h6m10.667427135s) [noise=true , address=145.79.227.34:4906 , piece_id=K3FDGUMH44QTYEQJ7CKHQZ7VLQWA3CLPZ47QU4PUL263TZV3U53A, seg_pos={0 0}, upload=465774]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw): protocol: expected piece hash; context canceled; EOF Dec 27 18:35:10.796354329 (531h6m10.667437375s) [noise=true , address=145.79.227.34:4906 , piece_id=K3FDGUMH44QTYEQJ7CKHQZ7VLQWA3CLPZ47QU4PUL263TZV3U53A, seg_pos={0 0}, upload=465774]: Long tail cancellation due to optimal threshold Dec 27 18:35:10.796460812 (531h6m10.667543857s) [noise=true , address=45.91.116.4:4940 , piece_id=5KUW3RSQGNU5N6PSQ6MFLPZAZHKAOVEZ7KBC5EEWNN2CZJIR4E7Q, seg_pos={0 0}, upload=465774]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: write tcp 188.166.225.170:45192->45.91.116.4:4940: use of closed network connection; write tcp 188.166.225.170:45192->45.91.116.4:4940: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45192->45.91.116.4:4940: use of closed network connection Dec 27 18:35:10.796470652 (531h6m10.667553697s) [noise=true , address=45.91.116.4:4940 , piece_id=5KUW3RSQGNU5N6PSQ6MFLPZAZHKAOVEZ7KBC5EEWNN2CZJIR4E7Q, seg_pos={0 0}, upload=465774]: Long tail cancellation due to optimal threshold Dec 27 18:35:10.796502903 (531h6m10.667585948s) [noise=true , address=77.90.31.14:4664 , piece_id=5QUNP6U67WATL7SK5DPAPUOMKMTE2M5ZFPS4VAFUQ3ZBPQWXW2SQ, seg_pos={0 0}, upload=465774]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12u52PWsW5g7XTky7sxS5mPFqpMD4AHCXFfera1WZyZF9xf6ma9): protocol: write tcp 188.166.225.170:44472->77.90.31.14:4664: use of closed network connection; write tcp 188.166.225.170:44472->77.90.31.14:4664: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44472->77.90.31.14:4664: use of closed network connection Dec 27 18:35:10.796511008 (531h6m10.667594053s) [noise=true , address=77.90.31.14:4664 , piece_id=5QUNP6U67WATL7SK5DPAPUOMKMTE2M5ZFPS4VAFUQ3ZBPQWXW2SQ, seg_pos={0 0}, upload=465774]: Long tail cancellation due to optimal threshold Dec 27 18:35:10.796532440 (531h6m10.667615485s) [seg_pos={0 0}, upload=465774]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 18:35:10.796676383 (531h6m10.667759429s) [upload=465774]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 18:35:10.796684238 (531h6m10.667767283s) [upload=465774]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 18:35:11.082699201 (531h6m10.953782246s) [upload=465774]: Done uploading segments... Dec 27 18:35:11.082746491 (531h6m10.953829537s) [upload=465774]: Upload finished. err: <nil> Duration: 30.002184802s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [XX....X...X.X........X.......X..X.XXXXXXX...X.........XXXX..] duration: 6.022083992s trace: https://jaeger.dev.storj.io/trace/77c06c284613bcdd FAIL (2025-12-27 18:34:44 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m, Piece ID: GQT4KT6DXZIUP3RBB5CIFT5LULTJZSEXVUMQDNK3SDU5D2OYZL2Q) context deadline exceeded; piecestore: (Node ID: 155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, Piece ID: QQBSXSDLRBULRGZ5ZDJJ4N3PQVHSB5IUNEXOVMBBPTA2F5DGH3OQ) context deadline exceeded ======LOGS====== .498693041s) [upload=465736]: Uploading segments... Dec 27 18:32:25.627698233 (531h3m25.498781268s) [upload=465736]: Got next segment. Inline: false Dec 27 18:32:25.627816168 (531h3m25.498899213s) [upload=465736]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 18:32:25.627823151 (531h3m25.498906187s) [upload=465736]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 18:32:25.874268727 (531h3m25.745351772s) [seg_pos={0 0}, upload=465736]: Begin upload segment... Dec 27 18:32:25.874324954 (531h3m25.745407999s) [seg_pos={0 0}, upload=465736]: Done begin upload segment. Dec 27 18:32:25.874351565 (531h3m25.7454346s) [noise=true , address=145.79.227.51:4409 , piece_id=BA5JBEPAEHZJUAVLZVX66JZVKHVDSTMQU3767BTH732YI57OHYRQ, seg_pos={0 0}, upload=465736]: Uploading piece... Dec 27 18:32:25.874375831 (531h3m25.745458876s) [noise=true , address=145.79.227.22:4510 , piece_id=TYDRPSG2QPLKSDETT6OUV4BXC4DYQSIR5Y2ZIWMFZDFMTGXTLRIA, seg_pos={0 0}, upload=465736]: Uploading piece... Dec 27 18:32:25.874436847 (531h3m25.745519882s) [noise=true , address=45.91.116.9:4264 , piece_id=GQT4KT6DXZIUP3RBB5CIFT5LULTJZSEXVUMQDNK3SDU5D2OYZL2Q, seg_pos={0 0}, upload=465736]: Uploading piece... Dec 27 18:32:25.875075614 (531h3m25.746158659s) [noise=true , address=77.90.31.19:4844 , piece_id=IQ6DTEAUK3ZMVVTVEXB65YXIA4R3WLWDBNCUZRZHFHRJ4K3LJM7A, seg_pos={0 0}, upload=465736]: Uploading piece... Dec 27 18:32:25.875509329 (531h3m25.746592375s) [noise=true , address=145.79.227.39:4013 , piece_id=Y2FSILI3NNR7Q5XXGDDLHHTKE2IRFU6O5AMLU547RZPXKWYQRWAA, seg_pos={0 0}, upload=465736]: Uploading piece... Dec 27 18:32:25.875941221 (531h3m25.747024267s) [noise=true , address=145.79.227.25:4618 , piece_id=ZCPAXIEOBJ3BJ2MSKK2EXBCCG523JMCFFVTLKWQ55UEBRMJZVVHA, seg_pos={0 0}, upload=465736]: Uploading piece... Dec 27 18:32:25.876358175 (531h3m25.747441221s) [noise=true , address=145.79.227.2:4114 , piece_id=KCTKDAQCZTBI3GRWEUBHK4GTLQQLCL3FRFNTKTYPHIVYBTO5I3AQ, seg_pos={0 0}, upload=465736]: Uploading piece... Dec 27 18:32:25.876859811 (531h3m25.747942856s) [noise=true , address=45.91.116.14:4189 , piece_id=WALZZPSLAPU6ZDNYRPN4FKWFND73GYWSUIXBYFZ3LA7GC2OZDCQQ, seg_pos={0 0}, upload=465736]: Uploading piece... Dec 27 18:32:25.878331714 (531h3m25.749414759s) [noise=true , address=145.79.227.26:4655 , piece_id=4YBF3DJ4BIH3GHFFG3YI4L4FYTVLDQEXTT5U6XEGHMG4S46BDNCQ, seg_pos={0 0}, upload=465736]: Uploading piece... Dec 27 18:32:25.878408840 (531h3m25.749491875s) [noise=true , address=145.79.227.47:4265 , piece_id=RCA6L5QL7ZBKHRFY7C6KEIBU6Y4EMIO2GEFDYF43BKJQ356JYAGQ, seg_pos={0 0}, upload=465736]: Uploading piece... Dec 27 18:32:25.880110090 (531h3m25.751193125s) [noise=true , address=145.79.227.38:4979 , piece_id=NDHM36NVO7M2SXJ4SVNYHWOBAO2DF2PBQDQRNU4N2ROMAGOOV4MQ, seg_pos={0 0}, upload=465736]: Uploading piece... Dec 27 18:32:25.880593010 (531h3m25.751676055s) [noise=true , address=145.79.227.15:4873 , piece_id=ADWK2XVAE7WOEZSD2FBLN4I5HEQDOMUNADKIG7BQBBBVVDIBADXQ, seg_pos={0 0}, upload=465736]: Uploading piece... Dec 27 18:32:25.881686892 (531h3m25.752769937s) [noise=true , address=45.91.116.12:4228 , piece_id=DSI5RZ5EC3GZ3VWSHPOYLXTP355TDAXJUPFTU6U77N3AVR63DO7Q, seg_pos={0 0}, upload=465736]: Uploading piece... Dec 27 18:32:25.883346833 (531h3m25.754429878s) [noise=true , address=77.90.31.14:4663 , piece_id=QQBSXSDLRBULRGZ5ZDJJ4N3PQVHSB5IUNEXOVMBBPTA2F5DGH3OQ, seg_pos={0 0}, upload=465736]: Uploading piece... Dec 27 18:32:25.883745723 (531h3m25.754828768s) [noise=true , address=145.79.227.28:4725 , piece_id=RZOCXC2TOJK75HJUKBYG6ZWJF5VKLAIZF6KFISEIDQIA6PEBCC5A, seg_pos={0 0}, upload=465736]: Uploading piece... Dec 27 18:32:25.886654291 (531h3m25.757737336s) [noise=true , address=77.90.31.21:4878 , piece_id=B6J6LYWK6W3DUWJJ7AWXZXVD3CAVK2PLZRV7LPGF3EDQCPLIQ6UA, seg_pos={0 0}, upload=465736]: Uploading piece... Dec 27 18:32:27.491958225 (531h3m27.36304126s) [upload=465736]: Next returned nil segment Dec 27 18:32:27.491973453 (531h3m27.363056488s) [upload=465736]: Waiting for error group managing segments... Dec 27 18:32:27.883820980 (531h3m27.754904025s) [noise=true , address=145.79.227.25:4618 , piece_id=ZCPAXIEOBJ3BJ2MSKK2EXBCCG523JMCFFVTLKWQ55UEBRMJZVVHA, seg_pos={0 0}, upload=465736]: Done uploading piece. err: <nil> Dec 27 18:32:27.884654017 (531h3m27.755737052s) [noise=true , address=145.79.227.38:4979 , piece_id=NDHM36NVO7M2SXJ4SVNYHWOBAO2DF2PBQDQRNU4N2ROMAGOOV4MQ, seg_pos={0 0}, upload=465736]: Done uploading piece. err: <nil> Dec 27 18:32:27.892675351 (531h3m27.763758386s) [noise=true , address=145.79.227.39:4013 , piece_id=Y2FSILI3NNR7Q5XXGDDLHHTKE2IRFU6O5AMLU547RZPXKWYQRWAA, seg_pos={0 0}, upload=465736]: Done uploading piece. err: <nil> Dec 27 18:32:27.896075837 (531h3m27.767158872s) [noise=true , address=145.79.227.47:4265 , piece_id=RCA6L5QL7ZBKHRFY7C6KEIBU6Y4EMIO2GEFDYF43BKJQ356JYAGQ, seg_pos={0 0}, upload=465736]: Done uploading piece. err: <nil> Dec 27 18:32:27.897659301 (531h3m27.768742336s) [noise=true , address=145.79.227.15:4873 , piece_id=ADWK2XVAE7WOEZSD2FBLN4I5HEQDOMUNADKIG7BQBBBVVDIBADXQ, seg_pos={0 0}, upload=465736]: Done uploading piece. err: <nil> Dec 27 18:32:27.898201484 (531h3m27.769284519s) [noise=true , address=145.79.227.22:4510 , piece_id=TYDRPSG2QPLKSDETT6OUV4BXC4DYQSIR5Y2ZIWMFZDFMTGXTLRIA, seg_pos={0 0}, upload=465736]: Done uploading piece. err: <nil> Dec 27 18:32:27.904575391 (531h3m27.775658426s) [noise=true , address=145.79.227.28:4725 , piece_id=RZOCXC2TOJK75HJUKBYG6ZWJF5VKLAIZF6KFISEIDQIA6PEBCC5A, seg_pos={0 0}, upload=465736]: Done uploading piece. err: <nil> Dec 27 18:32:27.912159122 (531h3m27.783242167s) [noise=true , address=145.79.227.26:4655 , piece_id=4YBF3DJ4BIH3GHFFG3YI4L4FYTVLDQEXTT5U6XEGHMG4S46BDNCQ, seg_pos={0 0}, upload=465736]: Done uploading piece. err: <nil> Dec 27 18:32:28.081094634 (531h3m27.952177669s) [noise=true , address=77.90.31.14:4663 , piece_id=QQBSXSDLRBULRGZ5ZDJJ4N3PQVHSB5IUNEXOVMBBPTA2F5DGH3OQ, seg_pos={0 0}, upload=465736]: Done uploading piece. err: <nil> Dec 27 18:32:28.148951857 (531h3m28.020034902s) [noise=true , address=45.91.116.14:4189 , piece_id=WALZZPSLAPU6ZDNYRPN4FKWFND73GYWSUIXBYFZ3LA7GC2OZDCQQ, seg_pos={0 0}, upload=465736]: Done uploading piece. err: <nil> Dec 27 18:32:28.152422566 (531h3m28.023505611s) [noise=true , address=45.91.116.12:4228 , piece_id=DSI5RZ5EC3GZ3VWSHPOYLXTP355TDAXJUPFTU6U77N3AVR63DO7Q, seg_pos={0 0}, upload=465736]: Done uploading piece. err: <nil> Dec 27 18:32:28.258556914 (531h3m28.129639949s) [noise=true , address=45.91.116.9:4264 , piece_id=GQT4KT6DXZIUP3RBB5CIFT5LULTJZSEXVUMQDNK3SDU5D2OYZL2Q, seg_pos={0 0}, upload=465736]: Done uploading piece. err: <nil> Dec 27 18:32:28.785386931 (531h3m28.656469976s) [noise=true , address=77.90.31.19:4844 , piece_id=IQ6DTEAUK3ZMVVTVEXB65YXIA4R3WLWDBNCUZRZHFHRJ4K3LJM7A, seg_pos={0 0}, upload=465736]: Done uploading piece. err: <nil> Dec 27 18:32:28.785411759 (531h3m28.656494804s) [seg_pos={0 0}, upload=465736]: Segment reached optimal threshold of 13 pieces. Dec 27 18:32:28.785707543 (531h3m28.656790588s) [noise=true , address=145.79.227.2:4114 , piece_id=KCTKDAQCZTBI3GRWEUBHK4GTLQQLCL3FRFNTKTYPHIVYBTO5I3AQ, seg_pos={0 0}, upload=465736]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN): protocol: write tcp 188.166.225.170:34096->145.79.227.2:4114: use of closed network connection; write tcp 188.166.225.170:34096->145.79.227.2:4114: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34096->145.79.227.2:4114: use of closed network connection Dec 27 18:32:28.785716920 (531h3m28.656799955s) [noise=true , address=145.79.227.2:4114 , piece_id=KCTKDAQCZTBI3GRWEUBHK4GTLQQLCL3FRFNTKTYPHIVYBTO5I3AQ, seg_pos={0 0}, upload=465736]: Long tail cancellation due to optimal threshold Dec 27 18:32:28.786070563 (531h3m28.657153609s) [noise=true , address=77.90.31.21:4878 , piece_id=B6J6LYWK6W3DUWJJ7AWXZXVD3CAVK2PLZRV7LPGF3EDQCPLIQ6UA, seg_pos={0 0}, upload=465736]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: expected piece hash; context canceled; EOF Dec 27 18:32:28.786079079 (531h3m28.657162115s) [noise=true , address=77.90.31.21:4878 , piece_id=B6J6LYWK6W3DUWJJ7AWXZXVD3CAVK2PLZRV7LPGF3EDQCPLIQ6UA, seg_pos={0 0}, upload=465736]: Long tail cancellation due to optimal threshold Dec 27 18:32:28.786208105 (531h3m28.65729114s) [noise=true , address=145.79.227.51:4409 , piece_id=BA5JBEPAEHZJUAVLZVX66JZVKHVDSTMQU3767BTH732YI57OHYRQ, seg_pos={0 0}, upload=465736]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UoCGA4FjE4yADPXL1oifSPJssPgSGh81REZHwGv2eu1Y1kEaj): protocol: write tcp 188.166.225.170:51222->145.79.227.51:4409: use of closed network connection; write tcp 188.166.225.170:51222->145.79.227.51:4409: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51222->145.79.227.51:4409: use of closed network connection Dec 27 18:32:28.786216872 (531h3m28.657299917s) [noise=true , address=145.79.227.51:4409 , piece_id=BA5JBEPAEHZJUAVLZVX66JZVKHVDSTMQU3767BTH732YI57OHYRQ, seg_pos={0 0}, upload=465736]: Long tail cancellation due to optimal threshold Dec 27 18:32:28.786237081 (531h3m28.657320126s) [seg_pos={0 0}, upload=465736]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 18:32:28.786323595 (531h3m28.65740664s) [upload=465736]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 18:32:28.786329556 (531h3m28.657412601s) [upload=465736]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 18:32:29.083652482 (531h3m28.954735517s) [upload=465736]: Done uploading segments... Dec 27 18:32:29.083676638 (531h3m28.954759683s) [upload=465736]: Upload finished. err: <nil> Duration: 6.022110243s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [.XX....X...X.X........X.......X..X.XXXXXXX...X.........XXXX.] duration: 14.356854779s trace: https://jaeger.dev.storj.io/trace/77c06c284613bcdd FAIL (2025-12-27 18:33:50 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m, Piece ID: GQT4KT6DXZIUP3RBB5CIFT5LULTJZSEXVUMQDNK3SDU5D2OYZL2Q) context deadline exceeded; piecestore: (Node ID: 155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, Piece ID: QQBSXSDLRBULRGZ5ZDJJ4N3PQVHSB5IUNEXOVMBBPTA2F5DGH3OQ) context deadline exceeded ======LOGS====== .498693041s) [upload=465736]: Uploading segments... Dec 27 18:32:25.627698233 (531h3m25.498781268s) [upload=465736]: Got next segment. Inline: false Dec 27 18:32:25.627816168 (531h3m25.498899213s) [upload=465736]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 18:32:25.627823151 (531h3m25.498906187s) [upload=465736]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 18:32:25.874268727 (531h3m25.745351772s) [seg_pos={0 0}, upload=465736]: Begin upload segment... Dec 27 18:32:25.874324954 (531h3m25.745407999s) [seg_pos={0 0}, upload=465736]: Done begin upload segment. Dec 27 18:32:25.874351565 (531h3m25.7454346s) [noise=true , address=145.79.227.51:4409 , piece_id=BA5JBEPAEHZJUAVLZVX66JZVKHVDSTMQU3767BTH732YI57OHYRQ, seg_pos={0 0}, upload=465736]: Uploading piece... Dec 27 18:32:25.874375831 (531h3m25.745458876s) [noise=true , address=145.79.227.22:4510 , piece_id=TYDRPSG2QPLKSDETT6OUV4BXC4DYQSIR5Y2ZIWMFZDFMTGXTLRIA, seg_pos={0 0}, upload=465736]: Uploading piece... Dec 27 18:32:25.874436847 (531h3m25.745519882s) [noise=true , address=45.91.116.9:4264 , piece_id=GQT4KT6DXZIUP3RBB5CIFT5LULTJZSEXVUMQDNK3SDU5D2OYZL2Q, seg_pos={0 0}, upload=465736]: Uploading piece... Dec 27 18:32:25.875075614 (531h3m25.746158659s) [noise=true , address=77.90.31.19:4844 , piece_id=IQ6DTEAUK3ZMVVTVEXB65YXIA4R3WLWDBNCUZRZHFHRJ4K3LJM7A, seg_pos={0 0}, upload=465736]: Uploading piece... Dec 27 18:32:25.875509329 (531h3m25.746592375s) [noise=true , address=145.79.227.39:4013 , piece_id=Y2FSILI3NNR7Q5XXGDDLHHTKE2IRFU6O5AMLU547RZPXKWYQRWAA, seg_pos={0 0}, upload=465736]: Uploading piece... Dec 27 18:32:25.875941221 (531h3m25.747024267s) [noise=true , address=145.79.227.25:4618 , piece_id=ZCPAXIEOBJ3BJ2MSKK2EXBCCG523JMCFFVTLKWQ55UEBRMJZVVHA, seg_pos={0 0}, upload=465736]: Uploading piece... Dec 27 18:32:25.876358175 (531h3m25.747441221s) [noise=true , address=145.79.227.2:4114 , piece_id=KCTKDAQCZTBI3GRWEUBHK4GTLQQLCL3FRFNTKTYPHIVYBTO5I3AQ, seg_pos={0 0}, upload=465736]: Uploading piece... Dec 27 18:32:25.876859811 (531h3m25.747942856s) [noise=true , address=45.91.116.14:4189 , piece_id=WALZZPSLAPU6ZDNYRPN4FKWFND73GYWSUIXBYFZ3LA7GC2OZDCQQ, seg_pos={0 0}, upload=465736]: Uploading piece... Dec 27 18:32:25.878331714 (531h3m25.749414759s) [noise=true , address=145.79.227.26:4655 , piece_id=4YBF3DJ4BIH3GHFFG3YI4L4FYTVLDQEXTT5U6XEGHMG4S46BDNCQ, seg_pos={0 0}, upload=465736]: Uploading piece... Dec 27 18:32:25.878408840 (531h3m25.749491875s) [noise=true , address=145.79.227.47:4265 , piece_id=RCA6L5QL7ZBKHRFY7C6KEIBU6Y4EMIO2GEFDYF43BKJQ356JYAGQ, seg_pos={0 0}, upload=465736]: Uploading piece... Dec 27 18:32:25.880110090 (531h3m25.751193125s) [noise=true , address=145.79.227.38:4979 , piece_id=NDHM36NVO7M2SXJ4SVNYHWOBAO2DF2PBQDQRNU4N2ROMAGOOV4MQ, seg_pos={0 0}, upload=465736]: Uploading piece... Dec 27 18:32:25.880593010 (531h3m25.751676055s) [noise=true , address=145.79.227.15:4873 , piece_id=ADWK2XVAE7WOEZSD2FBLN4I5HEQDOMUNADKIG7BQBBBVVDIBADXQ, seg_pos={0 0}, upload=465736]: Uploading piece... Dec 27 18:32:25.881686892 (531h3m25.752769937s) [noise=true , address=45.91.116.12:4228 , piece_id=DSI5RZ5EC3GZ3VWSHPOYLXTP355TDAXJUPFTU6U77N3AVR63DO7Q, seg_pos={0 0}, upload=465736]: Uploading piece... Dec 27 18:32:25.883346833 (531h3m25.754429878s) [noise=true , address=77.90.31.14:4663 , piece_id=QQBSXSDLRBULRGZ5ZDJJ4N3PQVHSB5IUNEXOVMBBPTA2F5DGH3OQ, seg_pos={0 0}, upload=465736]: Uploading piece... Dec 27 18:32:25.883745723 (531h3m25.754828768s) [noise=true , address=145.79.227.28:4725 , piece_id=RZOCXC2TOJK75HJUKBYG6ZWJF5VKLAIZF6KFISEIDQIA6PEBCC5A, seg_pos={0 0}, upload=465736]: Uploading piece... Dec 27 18:32:25.886654291 (531h3m25.757737336s) [noise=true , address=77.90.31.21:4878 , piece_id=B6J6LYWK6W3DUWJJ7AWXZXVD3CAVK2PLZRV7LPGF3EDQCPLIQ6UA, seg_pos={0 0}, upload=465736]: Uploading piece... Dec 27 18:32:27.491958225 (531h3m27.36304126s) [upload=465736]: Next returned nil segment Dec 27 18:32:27.491973453 (531h3m27.363056488s) [upload=465736]: Waiting for error group managing segments... Dec 27 18:32:27.883820980 (531h3m27.754904025s) [noise=true , address=145.79.227.25:4618 , piece_id=ZCPAXIEOBJ3BJ2MSKK2EXBCCG523JMCFFVTLKWQ55UEBRMJZVVHA, seg_pos={0 0}, upload=465736]: Done uploading piece. err: <nil> Dec 27 18:32:27.884654017 (531h3m27.755737052s) [noise=true , address=145.79.227.38:4979 , piece_id=NDHM36NVO7M2SXJ4SVNYHWOBAO2DF2PBQDQRNU4N2ROMAGOOV4MQ, seg_pos={0 0}, upload=465736]: Done uploading piece. err: <nil> Dec 27 18:32:27.892675351 (531h3m27.763758386s) [noise=true , address=145.79.227.39:4013 , piece_id=Y2FSILI3NNR7Q5XXGDDLHHTKE2IRFU6O5AMLU547RZPXKWYQRWAA, seg_pos={0 0}, upload=465736]: Done uploading piece. err: <nil> Dec 27 18:32:27.896075837 (531h3m27.767158872s) [noise=true , address=145.79.227.47:4265 , piece_id=RCA6L5QL7ZBKHRFY7C6KEIBU6Y4EMIO2GEFDYF43BKJQ356JYAGQ, seg_pos={0 0}, upload=465736]: Done uploading piece. err: <nil> Dec 27 18:32:27.897659301 (531h3m27.768742336s) [noise=true , address=145.79.227.15:4873 , piece_id=ADWK2XVAE7WOEZSD2FBLN4I5HEQDOMUNADKIG7BQBBBVVDIBADXQ, seg_pos={0 0}, upload=465736]: Done uploading piece. err: <nil> Dec 27 18:32:27.898201484 (531h3m27.769284519s) [noise=true , address=145.79.227.22:4510 , piece_id=TYDRPSG2QPLKSDETT6OUV4BXC4DYQSIR5Y2ZIWMFZDFMTGXTLRIA, seg_pos={0 0}, upload=465736]: Done uploading piece. err: <nil> Dec 27 18:32:27.904575391 (531h3m27.775658426s) [noise=true , address=145.79.227.28:4725 , piece_id=RZOCXC2TOJK75HJUKBYG6ZWJF5VKLAIZF6KFISEIDQIA6PEBCC5A, seg_pos={0 0}, upload=465736]: Done uploading piece. err: <nil> Dec 27 18:32:27.912159122 (531h3m27.783242167s) [noise=true , address=145.79.227.26:4655 , piece_id=4YBF3DJ4BIH3GHFFG3YI4L4FYTVLDQEXTT5U6XEGHMG4S46BDNCQ, seg_pos={0 0}, upload=465736]: Done uploading piece. err: <nil> Dec 27 18:32:28.081094634 (531h3m27.952177669s) [noise=true , address=77.90.31.14:4663 , piece_id=QQBSXSDLRBULRGZ5ZDJJ4N3PQVHSB5IUNEXOVMBBPTA2F5DGH3OQ, seg_pos={0 0}, upload=465736]: Done uploading piece. err: <nil> Dec 27 18:32:28.148951857 (531h3m28.020034902s) [noise=true , address=45.91.116.14:4189 , piece_id=WALZZPSLAPU6ZDNYRPN4FKWFND73GYWSUIXBYFZ3LA7GC2OZDCQQ, seg_pos={0 0}, upload=465736]: Done uploading piece. err: <nil> Dec 27 18:32:28.152422566 (531h3m28.023505611s) [noise=true , address=45.91.116.12:4228 , piece_id=DSI5RZ5EC3GZ3VWSHPOYLXTP355TDAXJUPFTU6U77N3AVR63DO7Q, seg_pos={0 0}, upload=465736]: Done uploading piece. err: <nil> Dec 27 18:32:28.258556914 (531h3m28.129639949s) [noise=true , address=45.91.116.9:4264 , piece_id=GQT4KT6DXZIUP3RBB5CIFT5LULTJZSEXVUMQDNK3SDU5D2OYZL2Q, seg_pos={0 0}, upload=465736]: Done uploading piece. err: <nil> Dec 27 18:32:28.785386931 (531h3m28.656469976s) [noise=true , address=77.90.31.19:4844 , piece_id=IQ6DTEAUK3ZMVVTVEXB65YXIA4R3WLWDBNCUZRZHFHRJ4K3LJM7A, seg_pos={0 0}, upload=465736]: Done uploading piece. err: <nil> Dec 27 18:32:28.785411759 (531h3m28.656494804s) [seg_pos={0 0}, upload=465736]: Segment reached optimal threshold of 13 pieces. Dec 27 18:32:28.785707543 (531h3m28.656790588s) [noise=true , address=145.79.227.2:4114 , piece_id=KCTKDAQCZTBI3GRWEUBHK4GTLQQLCL3FRFNTKTYPHIVYBTO5I3AQ, seg_pos={0 0}, upload=465736]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN): protocol: write tcp 188.166.225.170:34096->145.79.227.2:4114: use of closed network connection; write tcp 188.166.225.170:34096->145.79.227.2:4114: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34096->145.79.227.2:4114: use of closed network connection Dec 27 18:32:28.785716920 (531h3m28.656799955s) [noise=true , address=145.79.227.2:4114 , piece_id=KCTKDAQCZTBI3GRWEUBHK4GTLQQLCL3FRFNTKTYPHIVYBTO5I3AQ, seg_pos={0 0}, upload=465736]: Long tail cancellation due to optimal threshold Dec 27 18:32:28.786070563 (531h3m28.657153609s) [noise=true , address=77.90.31.21:4878 , piece_id=B6J6LYWK6W3DUWJJ7AWXZXVD3CAVK2PLZRV7LPGF3EDQCPLIQ6UA, seg_pos={0 0}, upload=465736]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: expected piece hash; context canceled; EOF Dec 27 18:32:28.786079079 (531h3m28.657162115s) [noise=true , address=77.90.31.21:4878 , piece_id=B6J6LYWK6W3DUWJJ7AWXZXVD3CAVK2PLZRV7LPGF3EDQCPLIQ6UA, seg_pos={0 0}, upload=465736]: Long tail cancellation due to optimal threshold Dec 27 18:32:28.786208105 (531h3m28.65729114s) [noise=true , address=145.79.227.51:4409 , piece_id=BA5JBEPAEHZJUAVLZVX66JZVKHVDSTMQU3767BTH732YI57OHYRQ, seg_pos={0 0}, upload=465736]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UoCGA4FjE4yADPXL1oifSPJssPgSGh81REZHwGv2eu1Y1kEaj): protocol: write tcp 188.166.225.170:51222->145.79.227.51:4409: use of closed network connection; write tcp 188.166.225.170:51222->145.79.227.51:4409: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51222->145.79.227.51:4409: use of closed network connection Dec 27 18:32:28.786216872 (531h3m28.657299917s) [noise=true , address=145.79.227.51:4409 , piece_id=BA5JBEPAEHZJUAVLZVX66JZVKHVDSTMQU3767BTH732YI57OHYRQ, seg_pos={0 0}, upload=465736]: Long tail cancellation due to optimal threshold Dec 27 18:32:28.786237081 (531h3m28.657320126s) [seg_pos={0 0}, upload=465736]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 18:32:28.786323595 (531h3m28.65740664s) [upload=465736]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 18:32:28.786329556 (531h3m28.657412601s) [upload=465736]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 18:32:29.083652482 (531h3m28.954735517s) [upload=465736]: Done uploading segments... Dec 27 18:32:29.083676638 (531h3m28.954759683s) [upload=465736]: Upload finished. err: <nil> Duration: 14.35688141s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [..XX....X...X.X........X.......X..X.XXXXXXX...X.........XXXX] duration: 30.003396882s trace: https://jaeger.dev.storj.io/trace/77c06c284613bcdd FAIL (2025-12-27 18:32:54 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m, Piece ID: GQT4KT6DXZIUP3RBB5CIFT5LULTJZSEXVUMQDNK3SDU5D2OYZL2Q) context deadline exceeded; piecestore: (Node ID: 155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, Piece ID: QQBSXSDLRBULRGZ5ZDJJ4N3PQVHSB5IUNEXOVMBBPTA2F5DGH3OQ) context deadline exceeded ======LOGS====== .498693041s) [upload=465736]: Uploading segments... Dec 27 18:32:25.627698233 (531h3m25.498781268s) [upload=465736]: Got next segment. Inline: false Dec 27 18:32:25.627816168 (531h3m25.498899213s) [upload=465736]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 18:32:25.627823151 (531h3m25.498906187s) [upload=465736]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 18:32:25.874268727 (531h3m25.745351772s) [seg_pos={0 0}, upload=465736]: Begin upload segment... Dec 27 18:32:25.874324954 (531h3m25.745407999s) [seg_pos={0 0}, upload=465736]: Done begin upload segment. Dec 27 18:32:25.874351565 (531h3m25.7454346s) [noise=true , address=145.79.227.51:4409 , piece_id=BA5JBEPAEHZJUAVLZVX66JZVKHVDSTMQU3767BTH732YI57OHYRQ, seg_pos={0 0}, upload=465736]: Uploading piece... Dec 27 18:32:25.874375831 (531h3m25.745458876s) [noise=true , address=145.79.227.22:4510 , piece_id=TYDRPSG2QPLKSDETT6OUV4BXC4DYQSIR5Y2ZIWMFZDFMTGXTLRIA, seg_pos={0 0}, upload=465736]: Uploading piece... Dec 27 18:32:25.874436847 (531h3m25.745519882s) [noise=true , address=45.91.116.9:4264 , piece_id=GQT4KT6DXZIUP3RBB5CIFT5LULTJZSEXVUMQDNK3SDU5D2OYZL2Q, seg_pos={0 0}, upload=465736]: Uploading piece... Dec 27 18:32:25.875075614 (531h3m25.746158659s) [noise=true , address=77.90.31.19:4844 , piece_id=IQ6DTEAUK3ZMVVTVEXB65YXIA4R3WLWDBNCUZRZHFHRJ4K3LJM7A, seg_pos={0 0}, upload=465736]: Uploading piece... Dec 27 18:32:25.875509329 (531h3m25.746592375s) [noise=true , address=145.79.227.39:4013 , piece_id=Y2FSILI3NNR7Q5XXGDDLHHTKE2IRFU6O5AMLU547RZPXKWYQRWAA, seg_pos={0 0}, upload=465736]: Uploading piece... Dec 27 18:32:25.875941221 (531h3m25.747024267s) [noise=true , address=145.79.227.25:4618 , piece_id=ZCPAXIEOBJ3BJ2MSKK2EXBCCG523JMCFFVTLKWQ55UEBRMJZVVHA, seg_pos={0 0}, upload=465736]: Uploading piece... Dec 27 18:32:25.876358175 (531h3m25.747441221s) [noise=true , address=145.79.227.2:4114 , piece_id=KCTKDAQCZTBI3GRWEUBHK4GTLQQLCL3FRFNTKTYPHIVYBTO5I3AQ, seg_pos={0 0}, upload=465736]: Uploading piece... Dec 27 18:32:25.876859811 (531h3m25.747942856s) [noise=true , address=45.91.116.14:4189 , piece_id=WALZZPSLAPU6ZDNYRPN4FKWFND73GYWSUIXBYFZ3LA7GC2OZDCQQ, seg_pos={0 0}, upload=465736]: Uploading piece... Dec 27 18:32:25.878331714 (531h3m25.749414759s) [noise=true , address=145.79.227.26:4655 , piece_id=4YBF3DJ4BIH3GHFFG3YI4L4FYTVLDQEXTT5U6XEGHMG4S46BDNCQ, seg_pos={0 0}, upload=465736]: Uploading piece... Dec 27 18:32:25.878408840 (531h3m25.749491875s) [noise=true , address=145.79.227.47:4265 , piece_id=RCA6L5QL7ZBKHRFY7C6KEIBU6Y4EMIO2GEFDYF43BKJQ356JYAGQ, seg_pos={0 0}, upload=465736]: Uploading piece... Dec 27 18:32:25.880110090 (531h3m25.751193125s) [noise=true , address=145.79.227.38:4979 , piece_id=NDHM36NVO7M2SXJ4SVNYHWOBAO2DF2PBQDQRNU4N2ROMAGOOV4MQ, seg_pos={0 0}, upload=465736]: Uploading piece... Dec 27 18:32:25.880593010 (531h3m25.751676055s) [noise=true , address=145.79.227.15:4873 , piece_id=ADWK2XVAE7WOEZSD2FBLN4I5HEQDOMUNADKIG7BQBBBVVDIBADXQ, seg_pos={0 0}, upload=465736]: Uploading piece... Dec 27 18:32:25.881686892 (531h3m25.752769937s) [noise=true , address=45.91.116.12:4228 , piece_id=DSI5RZ5EC3GZ3VWSHPOYLXTP355TDAXJUPFTU6U77N3AVR63DO7Q, seg_pos={0 0}, upload=465736]: Uploading piece... Dec 27 18:32:25.883346833 (531h3m25.754429878s) [noise=true , address=77.90.31.14:4663 , piece_id=QQBSXSDLRBULRGZ5ZDJJ4N3PQVHSB5IUNEXOVMBBPTA2F5DGH3OQ, seg_pos={0 0}, upload=465736]: Uploading piece... Dec 27 18:32:25.883745723 (531h3m25.754828768s) [noise=true , address=145.79.227.28:4725 , piece_id=RZOCXC2TOJK75HJUKBYG6ZWJF5VKLAIZF6KFISEIDQIA6PEBCC5A, seg_pos={0 0}, upload=465736]: Uploading piece... Dec 27 18:32:25.886654291 (531h3m25.757737336s) [noise=true , address=77.90.31.21:4878 , piece_id=B6J6LYWK6W3DUWJJ7AWXZXVD3CAVK2PLZRV7LPGF3EDQCPLIQ6UA, seg_pos={0 0}, upload=465736]: Uploading piece... Dec 27 18:32:27.491958225 (531h3m27.36304126s) [upload=465736]: Next returned nil segment Dec 27 18:32:27.491973453 (531h3m27.363056488s) [upload=465736]: Waiting for error group managing segments... Dec 27 18:32:27.883820980 (531h3m27.754904025s) [noise=true , address=145.79.227.25:4618 , piece_id=ZCPAXIEOBJ3BJ2MSKK2EXBCCG523JMCFFVTLKWQ55UEBRMJZVVHA, seg_pos={0 0}, upload=465736]: Done uploading piece. err: <nil> Dec 27 18:32:27.884654017 (531h3m27.755737052s) [noise=true , address=145.79.227.38:4979 , piece_id=NDHM36NVO7M2SXJ4SVNYHWOBAO2DF2PBQDQRNU4N2ROMAGOOV4MQ, seg_pos={0 0}, upload=465736]: Done uploading piece. err: <nil> Dec 27 18:32:27.892675351 (531h3m27.763758386s) [noise=true , address=145.79.227.39:4013 , piece_id=Y2FSILI3NNR7Q5XXGDDLHHTKE2IRFU6O5AMLU547RZPXKWYQRWAA, seg_pos={0 0}, upload=465736]: Done uploading piece. err: <nil> Dec 27 18:32:27.896075837 (531h3m27.767158872s) [noise=true , address=145.79.227.47:4265 , piece_id=RCA6L5QL7ZBKHRFY7C6KEIBU6Y4EMIO2GEFDYF43BKJQ356JYAGQ, seg_pos={0 0}, upload=465736]: Done uploading piece. err: <nil> Dec 27 18:32:27.897659301 (531h3m27.768742336s) [noise=true , address=145.79.227.15:4873 , piece_id=ADWK2XVAE7WOEZSD2FBLN4I5HEQDOMUNADKIG7BQBBBVVDIBADXQ, seg_pos={0 0}, upload=465736]: Done uploading piece. err: <nil> Dec 27 18:32:27.898201484 (531h3m27.769284519s) [noise=true , address=145.79.227.22:4510 , piece_id=TYDRPSG2QPLKSDETT6OUV4BXC4DYQSIR5Y2ZIWMFZDFMTGXTLRIA, seg_pos={0 0}, upload=465736]: Done uploading piece. err: <nil> Dec 27 18:32:27.904575391 (531h3m27.775658426s) [noise=true , address=145.79.227.28:4725 , piece_id=RZOCXC2TOJK75HJUKBYG6ZWJF5VKLAIZF6KFISEIDQIA6PEBCC5A, seg_pos={0 0}, upload=465736]: Done uploading piece. err: <nil> Dec 27 18:32:27.912159122 (531h3m27.783242167s) [noise=true , address=145.79.227.26:4655 , piece_id=4YBF3DJ4BIH3GHFFG3YI4L4FYTVLDQEXTT5U6XEGHMG4S46BDNCQ, seg_pos={0 0}, upload=465736]: Done uploading piece. err: <nil> Dec 27 18:32:28.081094634 (531h3m27.952177669s) [noise=true , address=77.90.31.14:4663 , piece_id=QQBSXSDLRBULRGZ5ZDJJ4N3PQVHSB5IUNEXOVMBBPTA2F5DGH3OQ, seg_pos={0 0}, upload=465736]: Done uploading piece. err: <nil> Dec 27 18:32:28.148951857 (531h3m28.020034902s) [noise=true , address=45.91.116.14:4189 , piece_id=WALZZPSLAPU6ZDNYRPN4FKWFND73GYWSUIXBYFZ3LA7GC2OZDCQQ, seg_pos={0 0}, upload=465736]: Done uploading piece. err: <nil> Dec 27 18:32:28.152422566 (531h3m28.023505611s) [noise=true , address=45.91.116.12:4228 , piece_id=DSI5RZ5EC3GZ3VWSHPOYLXTP355TDAXJUPFTU6U77N3AVR63DO7Q, seg_pos={0 0}, upload=465736]: Done uploading piece. err: <nil> Dec 27 18:32:28.258556914 (531h3m28.129639949s) [noise=true , address=45.91.116.9:4264 , piece_id=GQT4KT6DXZIUP3RBB5CIFT5LULTJZSEXVUMQDNK3SDU5D2OYZL2Q, seg_pos={0 0}, upload=465736]: Done uploading piece. err: <nil> Dec 27 18:32:28.785386931 (531h3m28.656469976s) [noise=true , address=77.90.31.19:4844 , piece_id=IQ6DTEAUK3ZMVVTVEXB65YXIA4R3WLWDBNCUZRZHFHRJ4K3LJM7A, seg_pos={0 0}, upload=465736]: Done uploading piece. err: <nil> Dec 27 18:32:28.785411759 (531h3m28.656494804s) [seg_pos={0 0}, upload=465736]: Segment reached optimal threshold of 13 pieces. Dec 27 18:32:28.785707543 (531h3m28.656790588s) [noise=true , address=145.79.227.2:4114 , piece_id=KCTKDAQCZTBI3GRWEUBHK4GTLQQLCL3FRFNTKTYPHIVYBTO5I3AQ, seg_pos={0 0}, upload=465736]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN): protocol: write tcp 188.166.225.170:34096->145.79.227.2:4114: use of closed network connection; write tcp 188.166.225.170:34096->145.79.227.2:4114: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34096->145.79.227.2:4114: use of closed network connection Dec 27 18:32:28.785716920 (531h3m28.656799955s) [noise=true , address=145.79.227.2:4114 , piece_id=KCTKDAQCZTBI3GRWEUBHK4GTLQQLCL3FRFNTKTYPHIVYBTO5I3AQ, seg_pos={0 0}, upload=465736]: Long tail cancellation due to optimal threshold Dec 27 18:32:28.786070563 (531h3m28.657153609s) [noise=true , address=77.90.31.21:4878 , piece_id=B6J6LYWK6W3DUWJJ7AWXZXVD3CAVK2PLZRV7LPGF3EDQCPLIQ6UA, seg_pos={0 0}, upload=465736]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: expected piece hash; context canceled; EOF Dec 27 18:32:28.786079079 (531h3m28.657162115s) [noise=true , address=77.90.31.21:4878 , piece_id=B6J6LYWK6W3DUWJJ7AWXZXVD3CAVK2PLZRV7LPGF3EDQCPLIQ6UA, seg_pos={0 0}, upload=465736]: Long tail cancellation due to optimal threshold Dec 27 18:32:28.786208105 (531h3m28.65729114s) [noise=true , address=145.79.227.51:4409 , piece_id=BA5JBEPAEHZJUAVLZVX66JZVKHVDSTMQU3767BTH732YI57OHYRQ, seg_pos={0 0}, upload=465736]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UoCGA4FjE4yADPXL1oifSPJssPgSGh81REZHwGv2eu1Y1kEaj): protocol: write tcp 188.166.225.170:51222->145.79.227.51:4409: use of closed network connection; write tcp 188.166.225.170:51222->145.79.227.51:4409: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51222->145.79.227.51:4409: use of closed network connection Dec 27 18:32:28.786216872 (531h3m28.657299917s) [noise=true , address=145.79.227.51:4409 , piece_id=BA5JBEPAEHZJUAVLZVX66JZVKHVDSTMQU3767BTH732YI57OHYRQ, seg_pos={0 0}, upload=465736]: Long tail cancellation due to optimal threshold Dec 27 18:32:28.786237081 (531h3m28.657320126s) [seg_pos={0 0}, upload=465736]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 18:32:28.786323595 (531h3m28.65740664s) [upload=465736]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 18:32:28.786329556 (531h3m28.657412601s) [upload=465736]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 18:32:29.083652482 (531h3m28.954735517s) [upload=465736]: Done uploading segments... Dec 27 18:32:29.083676638 (531h3m28.954759683s) [upload=465736]: Upload finished. err: <nil> Duration: 30.003421268s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [X..XX....X...X.X........X.......X..X.XXXXXXX...X.........XXX] duration: 30.00316466s trace: https://jaeger.dev.storj.io/trace/18c9fa9df826a6a7 FAIL (2025-12-27 18:31:41 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk, Piece ID: AF6M2V6UF7PXJTE7JKEAG4IYDNN4ADRENKH6UED6KEJSW5U7A7QQ) context deadline exceeded; piecestore: (Node ID: 121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp, Piece ID: SCELXJGKMY67R2KJC3V6IQ4JD73V6Q5DQNVXMHWQKL2SKKEMVKUQ) context deadline exceeded ======LOGS====== Dec 27 18:31:12.241162786 (531h2m12.112245821s) [upload=465717]: Starting upload Dec 27 18:31:12.241220416 (531h2m12.112303461s) [upload=465717]: Done starting upload Dec 27 18:31:12.241302723 (531h2m12.112385758s) [upload=465717]: Uploading segments... Dec 27 18:31:12.241441787 (531h2m12.112524822s) [upload=465717]: Got next segment. Inline: false Dec 27 18:31:12.241486282 (531h2m12.112569327s) [upload=465717]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 18:31:12.241491301 (531h2m12.112574346s) [upload=465717]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 18:31:12.486978342 (531h2m12.358061387s) [seg_pos={0 0}, upload=465717]: Begin upload segment... Dec 27 18:31:12.487054757 (531h2m12.358137802s) [seg_pos={0 0}, upload=465717]: Done begin upload segment. Dec 27 18:31:12.487129960 (531h2m12.358213005s) [noise=true , address=145.79.227.4:4185 , piece_id=5PS232COVE6CDLHQI7YVHBQL6W43ZU7TFJGDF737XAAVEYUMW47A, seg_pos={0 0}, upload=465717]: Uploading piece... Dec 27 18:31:12.487137825 (531h2m12.35822086s) [noise=true , address=145.79.227.46:4603 , piece_id=NF7KPI3KJDK5JOI3WSZORE7T5PECXGARCPY7TKA6K2SOFDFEWVVA, seg_pos={0 0}, upload=465717]: Uploading piece... Dec 27 18:31:12.487957516 (531h2m12.359040561s) [noise=true , address=77.90.31.14:4661 , piece_id=W3VAJIGEYO75LYQW5F6DLCA2UWUN3X53H77VM62D6BJV73TWW6SQ, seg_pos={0 0}, upload=465717]: Uploading piece... Dec 27 18:31:12.488370352 (531h2m12.359453398s) [noise=true , address=77.90.31.10:4517 , piece_id=MSBXRZJWYO5P7EIW76IGPDP6BHGYHTQKI36IJF3AZX6WVO7UDS2A, seg_pos={0 0}, upload=465717]: Uploading piece... Dec 27 18:31:12.488746779 (531h2m12.359829825s) [noise=true , address=145.79.227.11:4732 , piece_id=2XM7I2G6ZFXF67KLXZQ5XDNUOWFWEGSMAJVVZML6DRDYUAHNIR7A, seg_pos={0 0}, upload=465717]: Uploading piece... Dec 27 18:31:12.489134337 (531h2m12.360217402s) [noise=true , address=145.79.227.52:4447 , piece_id=5W4GKTVPQRH55GLQ4BBPOUFHBIC632PFY5GNQ2FO4VM6T6XYMFDQ, seg_pos={0 0}, upload=465717]: Uploading piece... Dec 27 18:31:12.489997511 (531h2m12.361080556s) [noise=true , address=45.91.116.6:4976 , piece_id=SCELXJGKMY67R2KJC3V6IQ4JD73V6Q5DQNVXMHWQKL2SKKEMVKUQ, seg_pos={0 0}, upload=465717]: Uploading piece... Dec 27 18:31:12.490015996 (531h2m12.361099041s) [noise=true , address=145.79.227.17:4970 , piece_id=NMEN6FN3A42SCMLTNL2CUPBMJ6NYRZQHVCFQ2U46QSNCLQTMKUPQ, seg_pos={0 0}, upload=465717]: Uploading piece... Dec 27 18:31:12.490582604 (531h2m12.36166565s) [noise=true , address=145.79.227.45:4230 , piece_id=RLEBMRZB6DNUR6H4DPTPDN5KZWUFMDSXXUPD5TBZIQVUU6JJZG6A, seg_pos={0 0}, upload=465717]: Uploading piece... Dec 27 18:31:12.490982346 (531h2m12.362065381s) [noise=true , address=145.79.227.51:4409 , piece_id=X34PJCL5LG4W4FRT6ZOCGAJCPTQBTTTDVOK7A4N7F4SAXPRPBY3A, seg_pos={0 0}, upload=465717]: Uploading piece... Dec 27 18:31:12.490987897 (531h2m12.362070942s) [noise=true , address=145.79.227.43:4158 , piece_id=NJVDBXYTMMXEV4M3R5VJQ7FCOP2GSSNVDRZJW4IQNXC7Q4DQLGTA, seg_pos={0 0}, upload=465717]: Uploading piece... Dec 27 18:31:12.491383089 (531h2m12.362466124s) [noise=true , address=145.79.227.33:4869 , piece_id=5JHLQAQYWHRPID7HTMCJMTZS32Q5DWFQYVLZ5HSNXSMRIKVQASLA, seg_pos={0 0}, upload=465717]: Uploading piece... Dec 27 18:31:12.491443985 (531h2m12.36252703s) [noise=true , address=145.79.227.44:4193 , piece_id=7CLLX4XD7V4CDVYGV3QQEZD2THJ6MPF2NWOUHOV7HIJT523NMZZQ, seg_pos={0 0}, upload=465717]: Uploading piece... Dec 27 18:31:12.491791917 (531h2m12.362874973s) [noise=true , address=77.90.31.16:4736 , piece_id=AF6M2V6UF7PXJTE7JKEAG4IYDNN4ADRENKH6UED6KEJSW5U7A7QQ, seg_pos={0 0}, upload=465717]: Uploading piece... Dec 27 18:31:12.492192641 (531h2m12.363275776s) [noise=true , address=145.79.227.28:4725 , piece_id=MYRFVZYTQGKEFG4EOYYD2C7Q3VMMZ24PR2RA5QCNA6ZVIZXKJMUQ, seg_pos={0 0}, upload=465717]: Uploading piece... Dec 27 18:31:12.494145700 (531h2m12.365228745s) [noise=true , address=145.79.227.24:4581 , piece_id=MDEVVHI2CCFPHZ6HBFFDZK3W5FGLTICPMIJJHLHEXAQZPP5NUPHQ, seg_pos={0 0}, upload=465717]: Uploading piece... Dec 27 18:31:14.201427335 (531h2m14.07251038s) [upload=465717]: Next returned nil segment Dec 27 18:31:14.201491427 (531h2m14.072574492s) [upload=465717]: Waiting for error group managing segments... Dec 27 18:31:14.490956901 (531h2m14.362039946s) [noise=true , address=145.79.227.33:4869 , piece_id=5JHLQAQYWHRPID7HTMCJMTZS32Q5DWFQYVLZ5HSNXSMRIKVQASLA, seg_pos={0 0}, upload=465717]: Done uploading piece. err: <nil> Dec 27 18:31:14.502855628 (531h2m14.373938673s) [noise=true , address=145.79.227.24:4581 , piece_id=MDEVVHI2CCFPHZ6HBFFDZK3W5FGLTICPMIJJHLHEXAQZPP5NUPHQ, seg_pos={0 0}, upload=465717]: Done uploading piece. err: <nil> Dec 27 18:31:14.503101676 (531h2m14.374184722s) [noise=true , address=145.79.227.51:4409 , piece_id=X34PJCL5LG4W4FRT6ZOCGAJCPTQBTTTDVOK7A4N7F4SAXPRPBY3A, seg_pos={0 0}, upload=465717]: Done uploading piece. err: <nil> Dec 27 18:31:14.504753843 (531h2m14.375836888s) [noise=true , address=145.79.227.11:4732 , piece_id=2XM7I2G6ZFXF67KLXZQ5XDNUOWFWEGSMAJVVZML6DRDYUAHNIR7A, seg_pos={0 0}, upload=465717]: Done uploading piece. err: <nil> Dec 27 18:31:14.509392886 (531h2m14.380475921s) [noise=true , address=145.79.227.46:4603 , piece_id=NF7KPI3KJDK5JOI3WSZORE7T5PECXGARCPY7TKA6K2SOFDFEWVVA, seg_pos={0 0}, upload=465717]: Done uploading piece. err: <nil> Dec 27 18:31:14.517549268 (531h2m14.388632313s) [noise=true , address=145.79.227.17:4970 , piece_id=NMEN6FN3A42SCMLTNL2CUPBMJ6NYRZQHVCFQ2U46QSNCLQTMKUPQ, seg_pos={0 0}, upload=465717]: Done uploading piece. err: <nil> Dec 27 18:31:14.517866021 (531h2m14.388949066s) [noise=true , address=145.79.227.28:4725 , piece_id=MYRFVZYTQGKEFG4EOYYD2C7Q3VMMZ24PR2RA5QCNA6ZVIZXKJMUQ, seg_pos={0 0}, upload=465717]: Done uploading piece. err: <nil> Dec 27 18:31:14.523928384 (531h2m14.395011429s) [noise=true , address=145.79.227.45:4230 , piece_id=RLEBMRZB6DNUR6H4DPTPDN5KZWUFMDSXXUPD5TBZIQVUU6JJZG6A, seg_pos={0 0}, upload=465717]: Done uploading piece. err: <nil> Dec 27 18:31:14.532490268 (531h2m14.403573303s) [noise=true , address=145.79.227.4:4185 , piece_id=5PS232COVE6CDLHQI7YVHBQL6W43ZU7TFJGDF737XAAVEYUMW47A, seg_pos={0 0}, upload=465717]: Done uploading piece. err: <nil> Dec 27 18:31:14.548647844 (531h2m14.419730899s) [noise=true , address=145.79.227.52:4447 , piece_id=5W4GKTVPQRH55GLQ4BBPOUFHBIC632PFY5GNQ2FO4VM6T6XYMFDQ, seg_pos={0 0}, upload=465717]: Done uploading piece. err: <nil> Dec 27 18:31:14.654579307 (531h2m14.525662352s) [noise=true , address=77.90.31.16:4736 , piece_id=AF6M2V6UF7PXJTE7JKEAG4IYDNN4ADRENKH6UED6KEJSW5U7A7QQ, seg_pos={0 0}, upload=465717]: Done uploading piece. err: <nil> Dec 27 18:31:14.677271918 (531h2m14.548354953s) [noise=true , address=77.90.31.10:4517 , piece_id=MSBXRZJWYO5P7EIW76IGPDP6BHGYHTQKI36IJF3AZX6WVO7UDS2A, seg_pos={0 0}, upload=465717]: Done uploading piece. err: <nil> Dec 27 18:31:15.128556273 (531h2m14.999639308s) [noise=true , address=45.91.116.6:4976 , piece_id=SCELXJGKMY67R2KJC3V6IQ4JD73V6Q5DQNVXMHWQKL2SKKEMVKUQ, seg_pos={0 0}, upload=465717]: Done uploading piece. err: <nil> Dec 27 18:31:15.128593284 (531h2m14.999676329s) [seg_pos={0 0}, upload=465717]: Segment reached optimal threshold of 13 pieces. Dec 27 18:31:15.128833821 (531h2m14.999916856s) [noise=true , address=145.79.227.44:4193 , piece_id=7CLLX4XD7V4CDVYGV3QQEZD2THJ6MPF2NWOUHOV7HIJT523NMZZQ, seg_pos={0 0}, upload=465717]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8): protocol: expected piece hash; context canceled; EOF Dec 27 18:31:15.128845123 (531h2m14.999928158s) [noise=true , address=145.79.227.44:4193 , piece_id=7CLLX4XD7V4CDVYGV3QQEZD2THJ6MPF2NWOUHOV7HIJT523NMZZQ, seg_pos={0 0}, upload=465717]: Long tail cancellation due to optimal threshold Dec 27 18:31:15.129017692 (531h2m15.000100777s) [noise=true , address=145.79.227.43:4158 , piece_id=NJVDBXYTMMXEV4M3R5VJQ7FCOP2GSSNVDRZJW4IQNXC7Q4DQLGTA, seg_pos={0 0}, upload=465717]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vJZrSvPvwebw9V2tTbvkPEmLeLjryBUnq9TL6NwdPsHU9xcq2): protocol: expected piece hash; context canceled; EOF Dec 27 18:31:15.129041015 (531h2m15.00012405s) [noise=true , address=145.79.227.43:4158 , piece_id=NJVDBXYTMMXEV4M3R5VJQ7FCOP2GSSNVDRZJW4IQNXC7Q4DQLGTA, seg_pos={0 0}, upload=465717]: Long tail cancellation due to optimal threshold Dec 27 18:31:15.129115508 (531h2m15.000198543s) [noise=true , address=77.90.31.14:4661 , piece_id=W3VAJIGEYO75LYQW5F6DLCA2UWUN3X53H77VM62D6BJV73TWW6SQ, seg_pos={0 0}, upload=465717]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue): protocol: expected piece hash; context canceled; EOF Dec 27 18:31:15.129126489 (531h2m15.000209524s) [noise=true , address=77.90.31.14:4661 , piece_id=W3VAJIGEYO75LYQW5F6DLCA2UWUN3X53H77VM62D6BJV73TWW6SQ, seg_pos={0 0}, upload=465717]: Long tail cancellation due to optimal threshold Dec 27 18:31:15.129167527 (531h2m15.000250572s) [seg_pos={0 0}, upload=465717]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 18:31:15.129306551 (531h2m15.000389586s) [upload=465717]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 18:31:15.129313074 (531h2m15.000396109s) [upload=465717]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 18:31:15.428428953 (531h2m15.299511998s) [upload=465717]: Done uploading segments... Dec 27 18:31:15.428453580 (531h2m15.299536625s) [upload=465717]: Upload finished. err: <nil> Duration: 30.003189467s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [XX..XX....X...X.X........X.......X..X.XXXXXXX...X.........XX] duration: 30.004224688s trace: https://jaeger.dev.storj.io/trace/5aac1552988aaa76 FAIL (2025-12-27 18:30:30 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, Piece ID: 5EUX7MSOOUW6IZHPIPYCTHOVZSCBIADTOQTQREPN7TD75VK6ROTA) context deadline exceeded; piecestore: (Node ID: 12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy, Piece ID: QYG6XKO3CKHG6KUCTY7GQVINC2DRVWDEJE5QC33WFPKZ5UBJ5S3Q) context deadline exceeded ======LOGS====== Dec 27 18:30:00.748321107 (531h1m0.619404152s) [upload=465701]: Starting upload Dec 27 18:30:00.748401441 (531h1m0.619484476s) [upload=465701]: Done starting upload Dec 27 18:30:00.748633031 (531h1m0.619716066s) [upload=465701]: Uploading segments... Dec 27 18:30:00.748685241 (531h1m0.619768306s) [upload=465701]: Got next segment. Inline: false Dec 27 18:30:00.748721179 (531h1m0.619804225s) [upload=465701]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 18:30:00.748727201 (531h1m0.619810246s) [upload=465701]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 18:30:00.995831922 (531h1m0.866914957s) [seg_pos={0 0}, upload=465701]: Begin upload segment... Dec 27 18:30:00.995876637 (531h1m0.866959672s) [seg_pos={0 0}, upload=465701]: Done begin upload segment. Dec 27 18:30:00.995907626 (531h1m0.866990661s) [noise=true , address=145.79.227.33:4871 , piece_id=RKAN7KPKKLGXZIYSMVEPEG67CSKG653665W27WKZZJ3TJXWIZVTA, seg_pos={0 0}, upload=465701]: Uploading piece... Dec 27 18:30:00.995913848 (531h1m0.866996883s) [noise=true , address=145.79.227.10:4403 , piece_id=USDQ7IM6BIUHBOQ3DSSUB66BK2QG6QDAS4W7LMOTOU44TVT7LFEA, seg_pos={0 0}, upload=465701]: Uploading piece... Dec 27 18:30:00.996762554 (531h1m0.867845589s) [noise=true , address=45.91.116.13:4156 , piece_id=PQ5XPR6S64MBPUOO3OA5TQFXFBCD2GLMXAFV2DIKCIX4BGXTW5UQ, seg_pos={0 0}, upload=465701]: Uploading piece... Dec 27 18:30:00.997335555 (531h1m0.8684186s) [noise=true , address=145.79.227.41:4085 , piece_id=KWG2FDD6IQAJK7EBYXAAKSQIKDPV7WXDLFKC6Q5L63OLVVOYSBWQ, seg_pos={0 0}, upload=465701]: Uploading piece... Dec 27 18:30:00.997514305 (531h1m0.868597361s) [noise=true , address=145.79.227.14:4838 , piece_id=A2J73SQE374LQILPUINGDMWKBX43CDQ4GCKO6BVIHGCAKXPSACTA, seg_pos={0 0}, upload=465701]: Uploading piece... Dec 27 18:30:00.997645005 (531h1m0.86872804s) [noise=true , address=77.90.31.10:4518 , piece_id=5EUX7MSOOUW6IZHPIPYCTHOVZSCBIADTOQTQREPN7TD75VK6ROTA, seg_pos={0 0}, upload=465701]: Uploading piece... Dec 27 18:30:00.997974281 (531h1m0.869057326s) [noise=true , address=145.79.227.37:4566 , piece_id=FB2JIYCCUD7DAYMA77HHEKHJ4ETOZQEQGKSMFF5PGRZS2ZMWXOXA, seg_pos={0 0}, upload=465701]: Uploading piece... Dec 27 18:30:00.998306795 (531h1m0.86938983s) [noise=true , address=145.79.227.25:4618 , piece_id=FAS5PRM34HAV7M677ZIBT6T3CQQ6VV5WI4P6RF4GQ6PB775IC3WA, seg_pos={0 0}, upload=465701]: Uploading piece... Dec 27 18:30:00.998666569 (531h1m0.869749615s) [noise=true , address=145.79.227.9:4365 , piece_id=ZOD3S7HS6BEC3RVCHHE4CL7OIY32ESO76WBELTKJXU7J4AXJNFFQ, seg_pos={0 0}, upload=465701]: Uploading piece... Dec 27 18:30:00.998970889 (531h1m0.870053934s) [noise=true , address=45.91.116.12:4225 , piece_id=QYG6XKO3CKHG6KUCTY7GQVINC2DRVWDEJE5QC33WFPKZ5UBJ5S3Q, seg_pos={0 0}, upload=465701]: Uploading piece... Dec 27 18:30:00.999491020 (531h1m0.870574055s) [noise=true , address=45.91.116.5:4297 , piece_id=SSDKZVGCJRTMV2E52753PFMNZW5HJSTT5CWZDK7WEPCJVPAUYLFA, seg_pos={0 0}, upload=465701]: Uploading piece... Dec 27 18:30:00.999684578 (531h1m0.870767613s) [noise=true , address=145.79.227.18:4007 , piece_id=G7XMCZXLIKAIW3YPPWX27EHDV6TJOUJADQXZUDBXLH3N7CLCUKHA, seg_pos={0 0}, upload=465701]: Uploading piece... Dec 27 18:30:01.000071464 (531h1m0.871154519s) [noise=true , address=77.90.31.14:4664 , piece_id=L6EWIHSINFNBYSRSGXBSK6R4ZCKM27MKHNQAKN75JDKU4I67D3SQ, seg_pos={0 0}, upload=465701]: Uploading piece... Dec 27 18:30:01.000520760 (531h1m0.871603805s) [noise=true , address=145.79.227.39:4015 , piece_id=TXZC5HD6QQZXPEEKAAW6SUUOATHH55QUGDN65T4UMI7UZCIS2KHA, seg_pos={0 0}, upload=465701]: Uploading piece... Dec 27 18:30:01.000710632 (531h1m0.871793687s) [noise=true , address=77.90.31.21:4877 , piece_id=NXBNCEFP63NXUOHO5JSSEAO6HG4PCHRCQGMTILKQMOU4E3XBWZYQ, seg_pos={0 0}, upload=465701]: Uploading piece... Dec 27 18:30:01.001231083 (531h1m0.872314138s) [noise=true , address=145.79.227.5:4221 , piece_id=VSGIZHRBE6KQ3MFFINBMWJRCJYNKY4YUH5XMFFT2V2P6S4ZKPZ2Q, seg_pos={0 0}, upload=465701]: Uploading piece... Dec 27 18:30:02.673938103 (531h1m2.545021148s) [upload=465701]: Next returned nil segment Dec 27 18:30:02.673957680 (531h1m2.545040725s) [upload=465701]: Waiting for error group managing segments... Dec 27 18:30:03.061403025 (531h1m2.93248607s) [noise=true , address=145.79.227.37:4566 , piece_id=FB2JIYCCUD7DAYMA77HHEKHJ4ETOZQEQGKSMFF5PGRZS2ZMWXOXA, seg_pos={0 0}, upload=465701]: Done uploading piece. err: <nil> Dec 27 18:30:03.062026623 (531h1m2.933109668s) [noise=true , address=145.79.227.41:4085 , piece_id=KWG2FDD6IQAJK7EBYXAAKSQIKDPV7WXDLFKC6Q5L63OLVVOYSBWQ, seg_pos={0 0}, upload=465701]: Done uploading piece. err: <nil> Dec 27 18:30:03.076402427 (531h1m2.947485472s) [noise=true , address=145.79.227.14:4838 , piece_id=A2J73SQE374LQILPUINGDMWKBX43CDQ4GCKO6BVIHGCAKXPSACTA, seg_pos={0 0}, upload=465701]: Done uploading piece. err: <nil> Dec 27 18:30:03.082135252 (531h1m2.953218297s) [noise=true , address=145.79.227.10:4403 , piece_id=USDQ7IM6BIUHBOQ3DSSUB66BK2QG6QDAS4W7LMOTOU44TVT7LFEA, seg_pos={0 0}, upload=465701]: Done uploading piece. err: <nil> Dec 27 18:30:03.083830200 (531h1m2.954913255s) [noise=true , address=145.79.227.25:4618 , piece_id=FAS5PRM34HAV7M677ZIBT6T3CQQ6VV5WI4P6RF4GQ6PB775IC3WA, seg_pos={0 0}, upload=465701]: Done uploading piece. err: <nil> Dec 27 18:30:03.087014132 (531h1m2.958097197s) [noise=true , address=145.79.227.39:4015 , piece_id=TXZC5HD6QQZXPEEKAAW6SUUOATHH55QUGDN65T4UMI7UZCIS2KHA, seg_pos={0 0}, upload=465701]: Done uploading piece. err: <nil> Dec 27 18:30:03.088039986 (531h1m2.959123031s) [noise=true , address=145.79.227.18:4007 , piece_id=G7XMCZXLIKAIW3YPPWX27EHDV6TJOUJADQXZUDBXLH3N7CLCUKHA, seg_pos={0 0}, upload=465701]: Done uploading piece. err: <nil> Dec 27 18:30:03.101145882 (531h1m2.972228937s) [noise=true , address=145.79.227.9:4365 , piece_id=ZOD3S7HS6BEC3RVCHHE4CL7OIY32ESO76WBELTKJXU7J4AXJNFFQ, seg_pos={0 0}, upload=465701]: Done uploading piece. err: <nil> Dec 27 18:30:03.198429116 (531h1m3.069512171s) [noise=true , address=145.79.227.33:4871 , piece_id=RKAN7KPKKLGXZIYSMVEPEG67CSKG653665W27WKZZJ3TJXWIZVTA, seg_pos={0 0}, upload=465701]: Done uploading piece. err: <nil> Dec 27 18:30:03.273417959 (531h1m3.144501004s) [noise=true , address=77.90.31.10:4518 , piece_id=5EUX7MSOOUW6IZHPIPYCTHOVZSCBIADTOQTQREPN7TD75VK6ROTA, seg_pos={0 0}, upload=465701]: Done uploading piece. err: <nil> Dec 27 18:30:03.291514177 (531h1m3.162597232s) [noise=true , address=77.90.31.14:4664 , piece_id=L6EWIHSINFNBYSRSGXBSK6R4ZCKM27MKHNQAKN75JDKU4I67D3SQ, seg_pos={0 0}, upload=465701]: Done uploading piece. err: <nil> Dec 27 18:30:03.298402454 (531h1m3.169485499s) [noise=true , address=45.91.116.12:4225 , piece_id=QYG6XKO3CKHG6KUCTY7GQVINC2DRVWDEJE5QC33WFPKZ5UBJ5S3Q, seg_pos={0 0}, upload=465701]: Done uploading piece. err: <nil> Dec 27 18:30:03.359797911 (531h1m3.230880957s) [noise=true , address=77.90.31.21:4877 , piece_id=NXBNCEFP63NXUOHO5JSSEAO6HG4PCHRCQGMTILKQMOU4E3XBWZYQ, seg_pos={0 0}, upload=465701]: Done uploading piece. err: <nil> Dec 27 18:30:03.359827547 (531h1m3.230910592s) [seg_pos={0 0}, upload=465701]: Segment reached optimal threshold of 13 pieces. Dec 27 18:30:03.360083174 (531h1m3.23116623s) [noise=true , address=45.91.116.13:4156 , piece_id=PQ5XPR6S64MBPUOO3OA5TQFXFBCD2GLMXAFV2DIKCIX4BGXTW5UQ, seg_pos={0 0}, upload=465701]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: expected piece hash; context canceled; EOF Dec 27 18:30:03.360094866 (531h1m3.231177911s) [noise=true , address=45.91.116.13:4156 , piece_id=PQ5XPR6S64MBPUOO3OA5TQFXFBCD2GLMXAFV2DIKCIX4BGXTW5UQ, seg_pos={0 0}, upload=465701]: Long tail cancellation due to optimal threshold Dec 27 18:30:03.360474840 (531h1m3.231557895s) [noise=true , address=145.79.227.5:4221 , piece_id=VSGIZHRBE6KQ3MFFINBMWJRCJYNKY4YUH5XMFFT2V2P6S4ZKPZ2Q, seg_pos={0 0}, upload=465701]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR): protocol: expected piece hash; context canceled; EOF Dec 27 18:30:03.360485209 (531h1m3.231568255s) [noise=true , address=145.79.227.5:4221 , piece_id=VSGIZHRBE6KQ3MFFINBMWJRCJYNKY4YUH5XMFFT2V2P6S4ZKPZ2Q, seg_pos={0 0}, upload=465701]: Long tail cancellation due to optimal threshold Dec 27 18:30:03.360660984 (531h1m3.231744029s) [noise=true , address=45.91.116.5:4297 , piece_id=SSDKZVGCJRTMV2E52753PFMNZW5HJSTT5CWZDK7WEPCJVPAUYLFA, seg_pos={0 0}, upload=465701]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: expected piece hash; context canceled; EOF Dec 27 18:30:03.360671094 (531h1m3.231754129s) [noise=true , address=45.91.116.5:4297 , piece_id=SSDKZVGCJRTMV2E52753PFMNZW5HJSTT5CWZDK7WEPCJVPAUYLFA, seg_pos={0 0}, upload=465701]: Long tail cancellation due to optimal threshold Dec 27 18:30:03.360698546 (531h1m3.231781591s) [seg_pos={0 0}, upload=465701]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 18:30:03.360800019 (531h1m3.231883074s) [upload=465701]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 18:30:03.360807554 (531h1m3.231890599s) [upload=465701]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 18:30:03.644441543 (531h1m3.515524588s) [upload=465701]: Done uploading segments... Dec 27 18:30:03.644468425 (531h1m3.51555147s) [upload=465701]: Upload finished. err: <nil> Duration: 30.004252401s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [X...........................................................] duration: 3.595652169s trace: https://jaeger.dev.storj.io/trace/3bddfef0dfa96557 FAIL (2025-12-27 18:29:52 UTC) orbiter warning: context deadline exceeded Duration: 3.595671095s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [XXX..XX....X...X.X........X.......X..X.XXXXXXX...X.........X] duration: 30.002717853s trace: https://jaeger.dev.storj.io/trace/7dd759c97b4c0757 FAIL (2025-12-27 18:29:40 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, Piece ID: SVMVZ7XTIXVQLCUEOBYKPYBS3B735DJJ5WID4QQGFWWIS5MSLULQ) context deadline exceeded; piecestore: (Node ID: 1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, Piece ID: BTX5PTJDWGFVMJDSGFDVPGKRJG5D3XT25N2O3XX5MNQ5SBVQDE5Q) context deadline exceeded ======LOGS====== Dec 27 18:29:11.201244583 (531h0m11.072327618s) [upload=465690]: Starting upload Dec 27 18:29:11.201303445 (531h0m11.07238649s) [upload=465690]: Done starting upload Dec 27 18:29:11.201497744 (531h0m11.07258079s) [upload=465690]: Uploading segments... Dec 27 18:29:11.201548772 (531h0m11.072631827s) [upload=465690]: Got next segment. Inline: false Dec 27 18:29:11.201575683 (531h0m11.072658718s) [upload=465690]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 18:29:11.201579962 (531h0m11.072662997s) [upload=465690]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 18:29:11.446917667 (531h0m11.318000712s) [seg_pos={0 0}, upload=465690]: Begin upload segment... Dec 27 18:29:11.446986318 (531h0m11.318069363s) [seg_pos={0 0}, upload=465690]: Done begin upload segment. Dec 27 18:29:11.447045932 (531h0m11.318128987s) [noise=true , address=145.79.227.36:4519 , piece_id=GCEHLTPG7JAE7LJXNFIAQUVSOQXNKHCWM5XSYURVFQRQSWMW2H3Q, seg_pos={0 0}, upload=465690]: Uploading piece... Dec 27 18:29:11.447069076 (531h0m11.318152121s) [noise=true , address=145.79.227.11:4729 , piece_id=MBVTA3IKXV3HEXFLIF362ROULECHWBDIY2ZFALRBCMVQKS7EXCNA, seg_pos={0 0}, upload=465690]: Uploading piece... Dec 27 18:29:11.447199083 (531h0m11.318282128s) [noise=true , address=145.79.227.51:4409 , piece_id=JE7GRUEAZJ5YERU5UTTWPU7PD453WV3QO73KBRLIXM7PLZKPBYKA, seg_pos={0 0}, upload=465690]: Uploading piece... Dec 27 18:29:11.447620676 (531h0m11.318703711s) [noise=true , address=145.79.227.39:4015 , piece_id=JPZ2243MWXGOJXPG4VKAO3PIQKXBWDH3ZXJOH3JBIT64M7OTUOSQ, seg_pos={0 0}, upload=465690]: Uploading piece... Dec 27 18:29:11.448017091 (531h0m11.319100126s) [noise=true , address=145.79.227.37:4567 , piece_id=2LCXOJ7J2KGUDOKQDZ25LIQXIUMMCOEBVLXE6S6JBGMSYKZDTK2A, seg_pos={0 0}, upload=465690]: Uploading piece... Dec 27 18:29:11.448371436 (531h0m11.319454471s) [noise=true , address=145.79.227.28:4725 , piece_id=55ZRJ5KXBMUU4RXZANXSS3FBQK52QYUYQW6SVBLMQXQ7GMSCJJCA, seg_pos={0 0}, upload=465690]: Uploading piece... Dec 27 18:29:11.448714539 (531h0m11.319797574s) [noise=true , address=145.79.227.23:4546 , piece_id=QCK2MVYBKLD3FMAQDKG6GY7YRHYN75CCJNFO75F5EIEAJQQTMQAA, seg_pos={0 0}, upload=465690]: Uploading piece... Dec 27 18:29:11.448989262 (531h0m11.320072297s) [noise=true , address=145.79.227.32:4834 , piece_id=WBNN47URS7ZV6RS7ZDEAEZGSIOTF3NNGMYJ5BLEWU45MGMXO644Q, seg_pos={0 0}, upload=465690]: Uploading piece... Dec 27 18:29:11.449084233 (531h0m11.320167278s) [noise=true , address=145.79.227.34:4907 , piece_id=POFQB5NEJ6VILLEBFTRNG3IJYTNKBSIUAT5X4SGVQV5X7YIRCFUA, seg_pos={0 0}, upload=465690]: Uploading piece... Dec 27 18:29:11.449478614 (531h0m11.320561649s) [noise=true , address=77.90.31.15:4700 , piece_id=SVMVZ7XTIXVQLCUEOBYKPYBS3B735DJJ5WID4QQGFWWIS5MSLULQ, seg_pos={0 0}, upload=465690]: Uploading piece... Dec 27 18:29:11.449509503 (531h0m11.320592548s) [noise=true , address=145.79.227.50:4373 , piece_id=YWS7ALQY4WW6B342OV6FNAV65NTT2NOTLXCBW7V3EZGGO6XPFIFQ, seg_pos={0 0}, upload=465690]: Uploading piece... Dec 27 18:29:11.451116894 (531h0m11.322199929s) [noise=true , address=145.79.227.14:4838 , piece_id=OWM7XP7V4FHR7CHBJS4ENYYNPO33RTODEGRW32RXBDEBYVVFTVVQ, seg_pos={0 0}, upload=465690]: Uploading piece... Dec 27 18:29:11.451669195 (531h0m11.322752241s) [noise=true , address=77.90.31.11:4554 , piece_id=MG3ZZJPYV4VHTSEMZIQJ6I54DCEH3KK33ACKF66F3VJIUGA23YTQ, seg_pos={0 0}, upload=465690]: Uploading piece... Dec 27 18:29:11.454106296 (531h0m11.325189341s) [noise=true , address=45.91.116.14:4191 , piece_id=BTX5PTJDWGFVMJDSGFDVPGKRJG5D3XT25N2O3XX5MNQ5SBVQDE5Q, seg_pos={0 0}, upload=465690]: Uploading piece... Dec 27 18:29:11.455538714 (531h0m11.326621759s) [noise=true , address=145.79.227.49:4339 , piece_id=2WU24VSIN6V6L6DMH5GZILBETDON6FLPZ5FZGB3RQ3FIINOJEOQA, seg_pos={0 0}, upload=465690]: Uploading piece... Dec 27 18:29:11.456823580 (531h0m11.327906625s) [noise=true , address=45.91.116.9:4261 , piece_id=JAIZDO4EJV7N7P4AY4I3UOMYMI22HVD7FTAAZ42F6QXLRFW5TC5Q, seg_pos={0 0}, upload=465690]: Uploading piece... Dec 27 18:29:13.157309896 (531h0m13.028392941s) [upload=465690]: Next returned nil segment Dec 27 18:29:13.157324574 (531h0m13.028407609s) [upload=465690]: Waiting for error group managing segments... Dec 27 18:29:13.439491512 (531h0m13.310574547s) [noise=true , address=145.79.227.50:4373 , piece_id=YWS7ALQY4WW6B342OV6FNAV65NTT2NOTLXCBW7V3EZGGO6XPFIFQ, seg_pos={0 0}, upload=465690]: Done uploading piece. err: <nil> Dec 27 18:29:13.446129471 (531h0m13.317212506s) [noise=true , address=145.79.227.11:4729 , piece_id=MBVTA3IKXV3HEXFLIF362ROULECHWBDIY2ZFALRBCMVQKS7EXCNA, seg_pos={0 0}, upload=465690]: Done uploading piece. err: <nil> Dec 27 18:29:13.448688274 (531h0m13.319771319s) [noise=true , address=145.79.227.51:4409 , piece_id=JE7GRUEAZJ5YERU5UTTWPU7PD453WV3QO73KBRLIXM7PLZKPBYKA, seg_pos={0 0}, upload=465690]: Done uploading piece. err: <nil> Dec 27 18:29:13.452146659 (531h0m13.323229704s) [noise=true , address=145.79.227.39:4015 , piece_id=JPZ2243MWXGOJXPG4VKAO3PIQKXBWDH3ZXJOH3JBIT64M7OTUOSQ, seg_pos={0 0}, upload=465690]: Done uploading piece. err: <nil> Dec 27 18:29:13.460183964 (531h0m13.331266999s) [noise=true , address=145.79.227.23:4546 , piece_id=QCK2MVYBKLD3FMAQDKG6GY7YRHYN75CCJNFO75F5EIEAJQQTMQAA, seg_pos={0 0}, upload=465690]: Done uploading piece. err: <nil> Dec 27 18:29:13.468932072 (531h0m13.340015107s) [noise=true , address=145.79.227.28:4725 , piece_id=55ZRJ5KXBMUU4RXZANXSS3FBQK52QYUYQW6SVBLMQXQ7GMSCJJCA, seg_pos={0 0}, upload=465690]: Done uploading piece. err: <nil> Dec 27 18:29:13.469785857 (531h0m13.340868902s) [noise=true , address=145.79.227.36:4519 , piece_id=GCEHLTPG7JAE7LJXNFIAQUVSOQXNKHCWM5XSYURVFQRQSWMW2H3Q, seg_pos={0 0}, upload=465690]: Done uploading piece. err: <nil> Dec 27 18:29:13.481650389 (531h0m13.352733424s) [noise=true , address=145.79.227.37:4567 , piece_id=2LCXOJ7J2KGUDOKQDZ25LIQXIUMMCOEBVLXE6S6JBGMSYKZDTK2A, seg_pos={0 0}, upload=465690]: Done uploading piece. err: <nil> Dec 27 18:29:13.498795567 (531h0m13.369878612s) [noise=true , address=145.79.227.32:4834 , piece_id=WBNN47URS7ZV6RS7ZDEAEZGSIOTF3NNGMYJ5BLEWU45MGMXO644Q, seg_pos={0 0}, upload=465690]: Done uploading piece. err: <nil> Dec 27 18:29:13.649789487 (531h0m13.520872522s) [noise=true , address=77.90.31.15:4700 , piece_id=SVMVZ7XTIXVQLCUEOBYKPYBS3B735DJJ5WID4QQGFWWIS5MSLULQ, seg_pos={0 0}, upload=465690]: Done uploading piece. err: <nil> Dec 27 18:29:13.750424244 (531h0m13.621507289s) [noise=true , address=45.91.116.14:4191 , piece_id=BTX5PTJDWGFVMJDSGFDVPGKRJG5D3XT25N2O3XX5MNQ5SBVQDE5Q, seg_pos={0 0}, upload=465690]: Done uploading piece. err: <nil> Dec 27 18:29:13.752470150 (531h0m13.623553195s) [noise=true , address=145.79.227.14:4838 , piece_id=OWM7XP7V4FHR7CHBJS4ENYYNPO33RTODEGRW32RXBDEBYVVFTVVQ, seg_pos={0 0}, upload=465690]: Done uploading piece. err: <nil> Dec 27 18:29:13.763968274 (531h0m13.635051309s) [noise=true , address=77.90.31.11:4554 , piece_id=MG3ZZJPYV4VHTSEMZIQJ6I54DCEH3KK33ACKF66F3VJIUGA23YTQ, seg_pos={0 0}, upload=465690]: Done uploading piece. err: <nil> Dec 27 18:29:13.763996989 (531h0m13.635080034s) [seg_pos={0 0}, upload=465690]: Segment reached optimal threshold of 13 pieces. Dec 27 18:29:13.764239220 (531h0m13.635322265s) [noise=true , address=145.79.227.34:4907 , piece_id=POFQB5NEJ6VILLEBFTRNG3IJYTNKBSIUAT5X4SGVQV5X7YIRCFUA, seg_pos={0 0}, upload=465690]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS): protocol: expected piece hash; context canceled; EOF Dec 27 18:29:13.764251834 (531h0m13.635334879s) [noise=true , address=145.79.227.34:4907 , piece_id=POFQB5NEJ6VILLEBFTRNG3IJYTNKBSIUAT5X4SGVQV5X7YIRCFUA, seg_pos={0 0}, upload=465690]: Long tail cancellation due to optimal threshold Dec 27 18:29:13.764353908 (531h0m13.635436954s) [noise=true , address=145.79.227.49:4339 , piece_id=2WU24VSIN6V6L6DMH5GZILBETDON6FLPZ5FZGB3RQ3FIINOJEOQA, seg_pos={0 0}, upload=465690]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124ACRNWgJYWgCVsDkXchmT47DiUcgvtZcdGo2MzyVqZkPjyz1R): protocol: expected piece hash; context canceled; EOF Dec 27 18:29:13.764448829 (531h0m13.635531875s) [noise=true , address=145.79.227.49:4339 , piece_id=2WU24VSIN6V6L6DMH5GZILBETDON6FLPZ5FZGB3RQ3FIINOJEOQA, seg_pos={0 0}, upload=465690]: Long tail cancellation due to optimal threshold Dec 27 18:29:13.764597883 (531h0m13.635680928s) [noise=true , address=45.91.116.9:4261 , piece_id=JAIZDO4EJV7N7P4AY4I3UOMYMI22HVD7FTAAZ42F6QXLRFW5TC5Q, seg_pos={0 0}, upload=465690]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi): protocol: expected piece hash; context canceled; EOF Dec 27 18:29:13.764612061 (531h0m13.635695126s) [noise=true , address=45.91.116.9:4261 , piece_id=JAIZDO4EJV7N7P4AY4I3UOMYMI22HVD7FTAAZ42F6QXLRFW5TC5Q, seg_pos={0 0}, upload=465690]: Long tail cancellation due to optimal threshold Dec 27 18:29:13.764644432 (531h0m13.635727477s) [seg_pos={0 0}, upload=465690]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 18:29:13.764805138 (531h0m13.635888183s) [upload=465690]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 18:29:13.764813414 (531h0m13.635896459s) [upload=465690]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 18:29:14.053215415 (531h0m13.92429845s) [upload=465690]: Done uploading segments... Dec 27 18:29:14.053235483 (531h0m13.924318518s) [upload=465690]: Upload finished. err: <nil> Duration: 30.002750185s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.X..........................................................] duration: 3.52251362s trace: https://jaeger.dev.storj.io/trace/3bddfef0dfa96557 FAIL (2025-12-27 18:28:40 UTC) orbiter warning: context deadline exceeded Duration: 3.522536784s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [.XXX..XX....X...X.X........X.......X..X.XXXXXXX...X.........] duration: 6.163401353s trace: https://jaeger.dev.storj.io/trace/4981aef9a433490 FAIL (2025-12-27 18:28:10 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, Piece ID: VDRCSP2C5VIHZR6YLLR7WUGURUZOXC2G7MT56IE43PMVVXDD3HZQ) context deadline exceeded; piecestore: (Node ID: 12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW, Piece ID: NVFVBNP6PWOJR3AT6ZB6LQWZ2JACMX5YK377GY2RE7GDM5FDPJUA) context deadline exceeded ======LOGS====== Dec 27 18:21:25.144885992 (530h52m25.015969028s) [upload=465581]: Starting upload Dec 27 18:21:25.144948482 (530h52m25.016031517s) [upload=465581]: Done starting upload Dec 27 18:21:25.145039054 (530h52m25.016122089s) [upload=465581]: Uploading segments... Dec 27 18:21:25.145258050 (530h52m25.016341095s) [upload=465581]: Got next segment. Inline: false Dec 27 18:21:25.145296154 (530h52m25.016379189s) [upload=465581]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 18:21:25.145301123 (530h52m25.016384158s) [upload=465581]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 18:21:25.389658082 (530h52m25.260741127s) [seg_pos={0 0}, upload=465581]: Begin upload segment... Dec 27 18:21:25.389719219 (530h52m25.260802264s) [seg_pos={0 0}, upload=465581]: Done begin upload segment. Dec 27 18:21:25.389751040 (530h52m25.260834095s) [noise=true , address=145.79.227.16:4910 , piece_id=PKF7JCLQKWL7DBRFMZQOTB2RURKDFJ6NMS7YB3AJYDJ2SC633JOQ, seg_pos={0 0}, upload=465581]: Uploading piece... Dec 27 18:21:25.389827786 (530h52m25.260910821s) [noise=true , address=45.91.116.6:4974 , piece_id=VDRCSP2C5VIHZR6YLLR7WUGURUZOXC2G7MT56IE43PMVVXDD3HZQ, seg_pos={0 0}, upload=465581]: Uploading piece... Dec 27 18:21:25.389925111 (530h52m25.261008146s) [noise=true , address=145.79.227.46:4602 , piece_id=OGGOAL326XC6BS7ULRPAM322APXHYL6FPWWVK32LHVANMFGYJWWQ, seg_pos={0 0}, upload=465581]: Uploading piece... Dec 27 18:21:25.390583174 (530h52m25.261666239s) [noise=true , address=145.79.227.11:4732 , piece_id=WVWGLOP2LWAJXKXXMVSIXEXUMPYLU5UPROXM4XFGCDWYVBXTEKOA, seg_pos={0 0}, upload=465581]: Uploading piece... Dec 27 18:21:25.391844145 (530h52m25.26292719s) [noise=true , address=145.79.227.2:4113 , piece_id=B3FPIHAGBGSJN2I6BFIO6S2YPXSIHKF7ZBODNV6FGOI7DQRYTIQA, seg_pos={0 0}, upload=465581]: Uploading piece... Dec 27 18:21:25.392261891 (530h52m25.263344936s) [noise=true , address=145.79.227.4:4185 , piece_id=BSDWYICA7RMLTXX4URC3O6YC25FHVQ6ZRYL7OS2227KE3DUSWJDQ, seg_pos={0 0}, upload=465581]: Uploading piece... Dec 27 18:21:25.392656372 (530h52m25.263739407s) [noise=true , address=45.91.116.13:4156 , piece_id=Y5TLN2KADJ7KUBGC6LFHRWSOSH3E2N2WLWM2O2X6O26YRL7LTDQA, seg_pos={0 0}, upload=465581]: Uploading piece... Dec 27 18:21:25.393053067 (530h52m25.264136112s) [noise=true , address=45.91.116.4:4939 , piece_id=EMKOBZFOAMID6S27D4UMCJL4JKCH2KGR4KPUKXG4P43BIJIIOGVA, seg_pos={0 0}, upload=465581]: Uploading piece... Dec 27 18:21:25.393516079 (530h52m25.264599114s) [noise=true , address=45.91.116.14:4192 , piece_id=4FEHZOQCMKHKWPHV6DCOTEOP7IYE7LQ22PX2WLFZ47IXQZ3IS36Q, seg_pos={0 0}, upload=465581]: Uploading piece... Dec 27 18:21:25.394785246 (530h52m25.265868291s) [noise=true , address=145.79.227.10:4403 , piece_id=IZP3UFQYXDPVJQIMBMP6LWWZ6TR52FN755X7VYCFTLREMHOIJN3A, seg_pos={0 0}, upload=465581]: Uploading piece... Dec 27 18:21:25.395302331 (530h52m25.266385376s) [noise=true , address=45.91.116.5:4297 , piece_id=NVFVBNP6PWOJR3AT6ZB6LQWZ2JACMX5YK377GY2RE7GDM5FDPJUA, seg_pos={0 0}, upload=465581]: Uploading piece... Dec 27 18:21:25.396222633 (530h52m25.267305678s) [noise=true , address=45.91.116.8:4046 , piece_id=6JWU3XRTODVGYUZBLZTMJJZFKYNTAHXGTJYQ2R7KXQKUBR7QSVKA, seg_pos={0 0}, upload=465581]: Uploading piece... Dec 27 18:21:25.397111025 (530h52m25.26819407s) [noise=true , address=145.79.227.17:4971 , piece_id=UO6L7APQZ46TBPJ5XOUHI2YL3Y5YZ2VAYFAZHOY7CL5C6BEO5G7Q, seg_pos={0 0}, upload=465581]: Uploading piece... Dec 27 18:21:25.398240415 (530h52m25.26932346s) [noise=true , address=145.79.227.36:4517 , piece_id=RLIQWAGNKI5X6WGHYMN7FXPHWDYEVGWXLFMPN7IEZR4ZRDARFIBQ, seg_pos={0 0}, upload=465581]: Uploading piece... Dec 27 18:21:25.402100956 (530h52m25.273184001s) [noise=true , address=145.79.227.48:4301 , piece_id=KKULR64B2HUQ7P4GSUX2XKQKW4DB3GVTD3PDJUOUPU7TOQ6ZKTJA, seg_pos={0 0}, upload=465581]: Uploading piece... Dec 27 18:21:25.405336728 (530h52m25.276419773s) [noise=true , address=145.79.227.45:4231 , piece_id=UX45A3K3LKYEG4YU7BVZKMJWYVUL47C5MMNQRSI32WE4Y62ILAOQ, seg_pos={0 0}, upload=465581]: Uploading piece... Dec 27 18:21:27.033083485 (530h52m26.90416653s) [upload=465581]: Next returned nil segment Dec 27 18:21:27.033095617 (530h52m26.904178662s) [upload=465581]: Waiting for error group managing segments... Dec 27 18:21:27.422573865 (530h52m27.2936569s) [noise=true , address=145.79.227.11:4732 , piece_id=WVWGLOP2LWAJXKXXMVSIXEXUMPYLU5UPROXM4XFGCDWYVBXTEKOA, seg_pos={0 0}, upload=465581]: Done uploading piece. err: <nil> Dec 27 18:21:27.425425164 (530h52m27.296508199s) [noise=true , address=145.79.227.46:4602 , piece_id=OGGOAL326XC6BS7ULRPAM322APXHYL6FPWWVK32LHVANMFGYJWWQ, seg_pos={0 0}, upload=465581]: Done uploading piece. err: <nil> Dec 27 18:21:27.442445785 (530h52m27.313528821s) [noise=true , address=145.79.227.45:4231 , piece_id=UX45A3K3LKYEG4YU7BVZKMJWYVUL47C5MMNQRSI32WE4Y62ILAOQ, seg_pos={0 0}, upload=465581]: Done uploading piece. err: <nil> Dec 27 18:21:27.449930007 (530h52m27.321013062s) [noise=true , address=145.79.227.17:4971 , piece_id=UO6L7APQZ46TBPJ5XOUHI2YL3Y5YZ2VAYFAZHOY7CL5C6BEO5G7Q, seg_pos={0 0}, upload=465581]: Done uploading piece. err: <nil> Dec 27 18:21:27.450121371 (530h52m27.321204406s) [noise=true , address=145.79.227.10:4403 , piece_id=IZP3UFQYXDPVJQIMBMP6LWWZ6TR52FN755X7VYCFTLREMHOIJN3A, seg_pos={0 0}, upload=465581]: Done uploading piece. err: <nil> Dec 27 18:21:27.454633623 (530h52m27.325716658s) [noise=true , address=145.79.227.48:4301 , piece_id=KKULR64B2HUQ7P4GSUX2XKQKW4DB3GVTD3PDJUOUPU7TOQ6ZKTJA, seg_pos={0 0}, upload=465581]: Done uploading piece. err: <nil> Dec 27 18:21:27.455522034 (530h52m27.326605079s) [noise=true , address=145.79.227.2:4113 , piece_id=B3FPIHAGBGSJN2I6BFIO6S2YPXSIHKF7ZBODNV6FGOI7DQRYTIQA, seg_pos={0 0}, upload=465581]: Done uploading piece. err: <nil> Dec 27 18:21:27.456210736 (530h52m27.327293781s) [noise=true , address=145.79.227.4:4185 , piece_id=BSDWYICA7RMLTXX4URC3O6YC25FHVQ6ZRYL7OS2227KE3DUSWJDQ, seg_pos={0 0}, upload=465581]: Done uploading piece. err: <nil> Dec 27 18:21:27.462096273 (530h52m27.333179308s) [noise=true , address=145.79.227.36:4517 , piece_id=RLIQWAGNKI5X6WGHYMN7FXPHWDYEVGWXLFMPN7IEZR4ZRDARFIBQ, seg_pos={0 0}, upload=465581]: Done uploading piece. err: <nil> Dec 27 18:21:27.509391311 (530h52m27.380474346s) [noise=true , address=145.79.227.16:4910 , piece_id=PKF7JCLQKWL7DBRFMZQOTB2RURKDFJ6NMS7YB3AJYDJ2SC633JOQ, seg_pos={0 0}, upload=465581]: Done uploading piece. err: <nil> Dec 27 18:21:27.640310134 (530h52m27.511393169s) [noise=true , address=45.91.116.6:4974 , piece_id=VDRCSP2C5VIHZR6YLLR7WUGURUZOXC2G7MT56IE43PMVVXDD3HZQ, seg_pos={0 0}, upload=465581]: Done uploading piece. err: <nil> Dec 27 18:21:27.671350691 (530h52m27.542433736s) [noise=true , address=45.91.116.5:4297 , piece_id=NVFVBNP6PWOJR3AT6ZB6LQWZ2JACMX5YK377GY2RE7GDM5FDPJUA, seg_pos={0 0}, upload=465581]: Done uploading piece. err: <nil> Dec 27 18:21:27.671881432 (530h52m27.542964477s) [noise=true , address=45.91.116.13:4156 , piece_id=Y5TLN2KADJ7KUBGC6LFHRWSOSH3E2N2WLWM2O2X6O26YRL7LTDQA, seg_pos={0 0}, upload=465581]: Done uploading piece. err: <nil> Dec 27 18:21:27.671891021 (530h52m27.542974056s) [seg_pos={0 0}, upload=465581]: Segment reached optimal threshold of 13 pieces. Dec 27 18:21:27.672486244 (530h52m27.543569279s) [noise=true , address=45.91.116.4:4939 , piece_id=EMKOBZFOAMID6S27D4UMCJL4JKCH2KGR4KPUKXG4P43BIJIIOGVA, seg_pos={0 0}, upload=465581]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: read tcp 188.166.225.170:39392->45.91.116.4:4939: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 27 18:21:27.672494189 (530h52m27.543577234s) [noise=true , address=45.91.116.4:4939 , piece_id=EMKOBZFOAMID6S27D4UMCJL4JKCH2KGR4KPUKXG4P43BIJIIOGVA, seg_pos={0 0}, upload=465581]: Long tail cancellation due to optimal threshold Dec 27 18:21:27.672941320 (530h52m27.544024365s) [noise=true , address=45.91.116.14:4192 , piece_id=4FEHZOQCMKHKWPHV6DCOTEOP7IYE7LQ22PX2WLFZ47IXQZ3IS36Q, seg_pos={0 0}, upload=465581]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs): protocol: read tcp 188.166.225.170:49760->45.91.116.14:4192: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 27 18:21:27.672949005 (530h52m27.54403204s) [noise=true , address=45.91.116.14:4192 , piece_id=4FEHZOQCMKHKWPHV6DCOTEOP7IYE7LQ22PX2WLFZ47IXQZ3IS36Q, seg_pos={0 0}, upload=465581]: Long tail cancellation due to optimal threshold Dec 27 18:21:27.673176889 (530h52m27.544259924s) [noise=true , address=45.91.116.8:4046 , piece_id=6JWU3XRTODVGYUZBLZTMJJZFKYNTAHXGTJYQ2R7KXQKUBR7QSVKA, seg_pos={0 0}, upload=465581]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H): protocol: expected piece hash; context canceled; EOF Dec 27 18:21:27.673184854 (530h52m27.544267899s) [noise=true , address=45.91.116.8:4046 , piece_id=6JWU3XRTODVGYUZBLZTMJJZFKYNTAHXGTJYQ2R7KXQKUBR7QSVKA, seg_pos={0 0}, upload=465581]: Long tail cancellation due to optimal threshold Dec 27 18:21:27.673203058 (530h52m27.544286093s) [seg_pos={0 0}, upload=465581]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 18:21:27.673307677 (530h52m27.544390713s) [upload=465581]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 18:21:27.673312197 (530h52m27.544395232s) [upload=465581]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 18:21:27.963559300 (530h52m27.834642355s) [upload=465581]: Done uploading segments... Dec 27 18:21:27.963583115 (530h52m27.83466616s) [upload=465581]: Upload finished. err: <nil> Duration: 6.163439494s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..X.........................................................] duration: 3.434775392s trace: https://jaeger.dev.storj.io/trace/3bddfef0dfa96557 FAIL (2025-12-27 18:27:33 UTC) orbiter warning: context deadline exceeded Duration: 3.434857096s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [..XXX..XX....X...X.X........X.......X..X.XXXXXXX...X........] duration: 13.569064724s trace: https://jaeger.dev.storj.io/trace/4981aef9a433490 FAIL (2025-12-27 18:27:04 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, Piece ID: VDRCSP2C5VIHZR6YLLR7WUGURUZOXC2G7MT56IE43PMVVXDD3HZQ) context deadline exceeded; piecestore: (Node ID: 12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW, Piece ID: NVFVBNP6PWOJR3AT6ZB6LQWZ2JACMX5YK377GY2RE7GDM5FDPJUA) context deadline exceeded ======LOGS====== Dec 27 18:21:25.144885992 (530h52m25.015969028s) [upload=465581]: Starting upload Dec 27 18:21:25.144948482 (530h52m25.016031517s) [upload=465581]: Done starting upload Dec 27 18:21:25.145039054 (530h52m25.016122089s) [upload=465581]: Uploading segments... Dec 27 18:21:25.145258050 (530h52m25.016341095s) [upload=465581]: Got next segment. Inline: false Dec 27 18:21:25.145296154 (530h52m25.016379189s) [upload=465581]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 18:21:25.145301123 (530h52m25.016384158s) [upload=465581]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 18:21:25.389658082 (530h52m25.260741127s) [seg_pos={0 0}, upload=465581]: Begin upload segment... Dec 27 18:21:25.389719219 (530h52m25.260802264s) [seg_pos={0 0}, upload=465581]: Done begin upload segment. Dec 27 18:21:25.389751040 (530h52m25.260834095s) [noise=true , address=145.79.227.16:4910 , piece_id=PKF7JCLQKWL7DBRFMZQOTB2RURKDFJ6NMS7YB3AJYDJ2SC633JOQ, seg_pos={0 0}, upload=465581]: Uploading piece... Dec 27 18:21:25.389827786 (530h52m25.260910821s) [noise=true , address=45.91.116.6:4974 , piece_id=VDRCSP2C5VIHZR6YLLR7WUGURUZOXC2G7MT56IE43PMVVXDD3HZQ, seg_pos={0 0}, upload=465581]: Uploading piece... Dec 27 18:21:25.389925111 (530h52m25.261008146s) [noise=true , address=145.79.227.46:4602 , piece_id=OGGOAL326XC6BS7ULRPAM322APXHYL6FPWWVK32LHVANMFGYJWWQ, seg_pos={0 0}, upload=465581]: Uploading piece... Dec 27 18:21:25.390583174 (530h52m25.261666239s) [noise=true , address=145.79.227.11:4732 , piece_id=WVWGLOP2LWAJXKXXMVSIXEXUMPYLU5UPROXM4XFGCDWYVBXTEKOA, seg_pos={0 0}, upload=465581]: Uploading piece... Dec 27 18:21:25.391844145 (530h52m25.26292719s) [noise=true , address=145.79.227.2:4113 , piece_id=B3FPIHAGBGSJN2I6BFIO6S2YPXSIHKF7ZBODNV6FGOI7DQRYTIQA, seg_pos={0 0}, upload=465581]: Uploading piece... Dec 27 18:21:25.392261891 (530h52m25.263344936s) [noise=true , address=145.79.227.4:4185 , piece_id=BSDWYICA7RMLTXX4URC3O6YC25FHVQ6ZRYL7OS2227KE3DUSWJDQ, seg_pos={0 0}, upload=465581]: Uploading piece... Dec 27 18:21:25.392656372 (530h52m25.263739407s) [noise=true , address=45.91.116.13:4156 , piece_id=Y5TLN2KADJ7KUBGC6LFHRWSOSH3E2N2WLWM2O2X6O26YRL7LTDQA, seg_pos={0 0}, upload=465581]: Uploading piece... Dec 27 18:21:25.393053067 (530h52m25.264136112s) [noise=true , address=45.91.116.4:4939 , piece_id=EMKOBZFOAMID6S27D4UMCJL4JKCH2KGR4KPUKXG4P43BIJIIOGVA, seg_pos={0 0}, upload=465581]: Uploading piece... Dec 27 18:21:25.393516079 (530h52m25.264599114s) [noise=true , address=45.91.116.14:4192 , piece_id=4FEHZOQCMKHKWPHV6DCOTEOP7IYE7LQ22PX2WLFZ47IXQZ3IS36Q, seg_pos={0 0}, upload=465581]: Uploading piece... Dec 27 18:21:25.394785246 (530h52m25.265868291s) [noise=true , address=145.79.227.10:4403 , piece_id=IZP3UFQYXDPVJQIMBMP6LWWZ6TR52FN755X7VYCFTLREMHOIJN3A, seg_pos={0 0}, upload=465581]: Uploading piece... Dec 27 18:21:25.395302331 (530h52m25.266385376s) [noise=true , address=45.91.116.5:4297 , piece_id=NVFVBNP6PWOJR3AT6ZB6LQWZ2JACMX5YK377GY2RE7GDM5FDPJUA, seg_pos={0 0}, upload=465581]: Uploading piece... Dec 27 18:21:25.396222633 (530h52m25.267305678s) [noise=true , address=45.91.116.8:4046 , piece_id=6JWU3XRTODVGYUZBLZTMJJZFKYNTAHXGTJYQ2R7KXQKUBR7QSVKA, seg_pos={0 0}, upload=465581]: Uploading piece... Dec 27 18:21:25.397111025 (530h52m25.26819407s) [noise=true , address=145.79.227.17:4971 , piece_id=UO6L7APQZ46TBPJ5XOUHI2YL3Y5YZ2VAYFAZHOY7CL5C6BEO5G7Q, seg_pos={0 0}, upload=465581]: Uploading piece... Dec 27 18:21:25.398240415 (530h52m25.26932346s) [noise=true , address=145.79.227.36:4517 , piece_id=RLIQWAGNKI5X6WGHYMN7FXPHWDYEVGWXLFMPN7IEZR4ZRDARFIBQ, seg_pos={0 0}, upload=465581]: Uploading piece... Dec 27 18:21:25.402100956 (530h52m25.273184001s) [noise=true , address=145.79.227.48:4301 , piece_id=KKULR64B2HUQ7P4GSUX2XKQKW4DB3GVTD3PDJUOUPU7TOQ6ZKTJA, seg_pos={0 0}, upload=465581]: Uploading piece... Dec 27 18:21:25.405336728 (530h52m25.276419773s) [noise=true , address=145.79.227.45:4231 , piece_id=UX45A3K3LKYEG4YU7BVZKMJWYVUL47C5MMNQRSI32WE4Y62ILAOQ, seg_pos={0 0}, upload=465581]: Uploading piece... Dec 27 18:21:27.033083485 (530h52m26.90416653s) [upload=465581]: Next returned nil segment Dec 27 18:21:27.033095617 (530h52m26.904178662s) [upload=465581]: Waiting for error group managing segments... Dec 27 18:21:27.422573865 (530h52m27.2936569s) [noise=true , address=145.79.227.11:4732 , piece_id=WVWGLOP2LWAJXKXXMVSIXEXUMPYLU5UPROXM4XFGCDWYVBXTEKOA, seg_pos={0 0}, upload=465581]: Done uploading piece. err: <nil> Dec 27 18:21:27.425425164 (530h52m27.296508199s) [noise=true , address=145.79.227.46:4602 , piece_id=OGGOAL326XC6BS7ULRPAM322APXHYL6FPWWVK32LHVANMFGYJWWQ, seg_pos={0 0}, upload=465581]: Done uploading piece. err: <nil> Dec 27 18:21:27.442445785 (530h52m27.313528821s) [noise=true , address=145.79.227.45:4231 , piece_id=UX45A3K3LKYEG4YU7BVZKMJWYVUL47C5MMNQRSI32WE4Y62ILAOQ, seg_pos={0 0}, upload=465581]: Done uploading piece. err: <nil> Dec 27 18:21:27.449930007 (530h52m27.321013062s) [noise=true , address=145.79.227.17:4971 , piece_id=UO6L7APQZ46TBPJ5XOUHI2YL3Y5YZ2VAYFAZHOY7CL5C6BEO5G7Q, seg_pos={0 0}, upload=465581]: Done uploading piece. err: <nil> Dec 27 18:21:27.450121371 (530h52m27.321204406s) [noise=true , address=145.79.227.10:4403 , piece_id=IZP3UFQYXDPVJQIMBMP6LWWZ6TR52FN755X7VYCFTLREMHOIJN3A, seg_pos={0 0}, upload=465581]: Done uploading piece. err: <nil> Dec 27 18:21:27.454633623 (530h52m27.325716658s) [noise=true , address=145.79.227.48:4301 , piece_id=KKULR64B2HUQ7P4GSUX2XKQKW4DB3GVTD3PDJUOUPU7TOQ6ZKTJA, seg_pos={0 0}, upload=465581]: Done uploading piece. err: <nil> Dec 27 18:21:27.455522034 (530h52m27.326605079s) [noise=true , address=145.79.227.2:4113 , piece_id=B3FPIHAGBGSJN2I6BFIO6S2YPXSIHKF7ZBODNV6FGOI7DQRYTIQA, seg_pos={0 0}, upload=465581]: Done uploading piece. err: <nil> Dec 27 18:21:27.456210736 (530h52m27.327293781s) [noise=true , address=145.79.227.4:4185 , piece_id=BSDWYICA7RMLTXX4URC3O6YC25FHVQ6ZRYL7OS2227KE3DUSWJDQ, seg_pos={0 0}, upload=465581]: Done uploading piece. err: <nil> Dec 27 18:21:27.462096273 (530h52m27.333179308s) [noise=true , address=145.79.227.36:4517 , piece_id=RLIQWAGNKI5X6WGHYMN7FXPHWDYEVGWXLFMPN7IEZR4ZRDARFIBQ, seg_pos={0 0}, upload=465581]: Done uploading piece. err: <nil> Dec 27 18:21:27.509391311 (530h52m27.380474346s) [noise=true , address=145.79.227.16:4910 , piece_id=PKF7JCLQKWL7DBRFMZQOTB2RURKDFJ6NMS7YB3AJYDJ2SC633JOQ, seg_pos={0 0}, upload=465581]: Done uploading piece. err: <nil> Dec 27 18:21:27.640310134 (530h52m27.511393169s) [noise=true , address=45.91.116.6:4974 , piece_id=VDRCSP2C5VIHZR6YLLR7WUGURUZOXC2G7MT56IE43PMVVXDD3HZQ, seg_pos={0 0}, upload=465581]: Done uploading piece. err: <nil> Dec 27 18:21:27.671350691 (530h52m27.542433736s) [noise=true , address=45.91.116.5:4297 , piece_id=NVFVBNP6PWOJR3AT6ZB6LQWZ2JACMX5YK377GY2RE7GDM5FDPJUA, seg_pos={0 0}, upload=465581]: Done uploading piece. err: <nil> Dec 27 18:21:27.671881432 (530h52m27.542964477s) [noise=true , address=45.91.116.13:4156 , piece_id=Y5TLN2KADJ7KUBGC6LFHRWSOSH3E2N2WLWM2O2X6O26YRL7LTDQA, seg_pos={0 0}, upload=465581]: Done uploading piece. err: <nil> Dec 27 18:21:27.671891021 (530h52m27.542974056s) [seg_pos={0 0}, upload=465581]: Segment reached optimal threshold of 13 pieces. Dec 27 18:21:27.672486244 (530h52m27.543569279s) [noise=true , address=45.91.116.4:4939 , piece_id=EMKOBZFOAMID6S27D4UMCJL4JKCH2KGR4KPUKXG4P43BIJIIOGVA, seg_pos={0 0}, upload=465581]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: read tcp 188.166.225.170:39392->45.91.116.4:4939: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 27 18:21:27.672494189 (530h52m27.543577234s) [noise=true , address=45.91.116.4:4939 , piece_id=EMKOBZFOAMID6S27D4UMCJL4JKCH2KGR4KPUKXG4P43BIJIIOGVA, seg_pos={0 0}, upload=465581]: Long tail cancellation due to optimal threshold Dec 27 18:21:27.672941320 (530h52m27.544024365s) [noise=true , address=45.91.116.14:4192 , piece_id=4FEHZOQCMKHKWPHV6DCOTEOP7IYE7LQ22PX2WLFZ47IXQZ3IS36Q, seg_pos={0 0}, upload=465581]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs): protocol: read tcp 188.166.225.170:49760->45.91.116.14:4192: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 27 18:21:27.672949005 (530h52m27.54403204s) [noise=true , address=45.91.116.14:4192 , piece_id=4FEHZOQCMKHKWPHV6DCOTEOP7IYE7LQ22PX2WLFZ47IXQZ3IS36Q, seg_pos={0 0}, upload=465581]: Long tail cancellation due to optimal threshold Dec 27 18:21:27.673176889 (530h52m27.544259924s) [noise=true , address=45.91.116.8:4046 , piece_id=6JWU3XRTODVGYUZBLZTMJJZFKYNTAHXGTJYQ2R7KXQKUBR7QSVKA, seg_pos={0 0}, upload=465581]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H): protocol: expected piece hash; context canceled; EOF Dec 27 18:21:27.673184854 (530h52m27.544267899s) [noise=true , address=45.91.116.8:4046 , piece_id=6JWU3XRTODVGYUZBLZTMJJZFKYNTAHXGTJYQ2R7KXQKUBR7QSVKA, seg_pos={0 0}, upload=465581]: Long tail cancellation due to optimal threshold Dec 27 18:21:27.673203058 (530h52m27.544286093s) [seg_pos={0 0}, upload=465581]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 18:21:27.673307677 (530h52m27.544390713s) [upload=465581]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 18:21:27.673312197 (530h52m27.544395232s) [upload=465581]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 18:21:27.963559300 (530h52m27.834642355s) [upload=465581]: Done uploading segments... Dec 27 18:21:27.963583115 (530h52m27.83466616s) [upload=465581]: Upload finished. err: <nil> Duration: 13.569097467s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...X........................................................] duration: 3.558016323s trace: https://jaeger.dev.storj.io/trace/3bddfef0dfa96557 FAIL (2025-12-27 18:26:30 UTC) orbiter warning: context deadline exceeded Duration: 3.55807298s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [...XXX..XX....X...X.X........X.......X..X.XXXXXXX...X.......] duration: 15.249211185s trace: https://jaeger.dev.storj.io/trace/4981aef9a433490 FAIL (2025-12-27 18:25:53 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, Piece ID: VDRCSP2C5VIHZR6YLLR7WUGURUZOXC2G7MT56IE43PMVVXDD3HZQ) context deadline exceeded; piecestore: (Node ID: 12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW, Piece ID: NVFVBNP6PWOJR3AT6ZB6LQWZ2JACMX5YK377GY2RE7GDM5FDPJUA) context deadline exceeded ======LOGS====== Dec 27 18:21:25.144885992 (530h52m25.015969028s) [upload=465581]: Starting upload Dec 27 18:21:25.144948482 (530h52m25.016031517s) [upload=465581]: Done starting upload Dec 27 18:21:25.145039054 (530h52m25.016122089s) [upload=465581]: Uploading segments... Dec 27 18:21:25.145258050 (530h52m25.016341095s) [upload=465581]: Got next segment. Inline: false Dec 27 18:21:25.145296154 (530h52m25.016379189s) [upload=465581]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 18:21:25.145301123 (530h52m25.016384158s) [upload=465581]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 18:21:25.389658082 (530h52m25.260741127s) [seg_pos={0 0}, upload=465581]: Begin upload segment... Dec 27 18:21:25.389719219 (530h52m25.260802264s) [seg_pos={0 0}, upload=465581]: Done begin upload segment. Dec 27 18:21:25.389751040 (530h52m25.260834095s) [noise=true , address=145.79.227.16:4910 , piece_id=PKF7JCLQKWL7DBRFMZQOTB2RURKDFJ6NMS7YB3AJYDJ2SC633JOQ, seg_pos={0 0}, upload=465581]: Uploading piece... Dec 27 18:21:25.389827786 (530h52m25.260910821s) [noise=true , address=45.91.116.6:4974 , piece_id=VDRCSP2C5VIHZR6YLLR7WUGURUZOXC2G7MT56IE43PMVVXDD3HZQ, seg_pos={0 0}, upload=465581]: Uploading piece... Dec 27 18:21:25.389925111 (530h52m25.261008146s) [noise=true , address=145.79.227.46:4602 , piece_id=OGGOAL326XC6BS7ULRPAM322APXHYL6FPWWVK32LHVANMFGYJWWQ, seg_pos={0 0}, upload=465581]: Uploading piece... Dec 27 18:21:25.390583174 (530h52m25.261666239s) [noise=true , address=145.79.227.11:4732 , piece_id=WVWGLOP2LWAJXKXXMVSIXEXUMPYLU5UPROXM4XFGCDWYVBXTEKOA, seg_pos={0 0}, upload=465581]: Uploading piece... Dec 27 18:21:25.391844145 (530h52m25.26292719s) [noise=true , address=145.79.227.2:4113 , piece_id=B3FPIHAGBGSJN2I6BFIO6S2YPXSIHKF7ZBODNV6FGOI7DQRYTIQA, seg_pos={0 0}, upload=465581]: Uploading piece... Dec 27 18:21:25.392261891 (530h52m25.263344936s) [noise=true , address=145.79.227.4:4185 , piece_id=BSDWYICA7RMLTXX4URC3O6YC25FHVQ6ZRYL7OS2227KE3DUSWJDQ, seg_pos={0 0}, upload=465581]: Uploading piece... Dec 27 18:21:25.392656372 (530h52m25.263739407s) [noise=true , address=45.91.116.13:4156 , piece_id=Y5TLN2KADJ7KUBGC6LFHRWSOSH3E2N2WLWM2O2X6O26YRL7LTDQA, seg_pos={0 0}, upload=465581]: Uploading piece... Dec 27 18:21:25.393053067 (530h52m25.264136112s) [noise=true , address=45.91.116.4:4939 , piece_id=EMKOBZFOAMID6S27D4UMCJL4JKCH2KGR4KPUKXG4P43BIJIIOGVA, seg_pos={0 0}, upload=465581]: Uploading piece... Dec 27 18:21:25.393516079 (530h52m25.264599114s) [noise=true , address=45.91.116.14:4192 , piece_id=4FEHZOQCMKHKWPHV6DCOTEOP7IYE7LQ22PX2WLFZ47IXQZ3IS36Q, seg_pos={0 0}, upload=465581]: Uploading piece... Dec 27 18:21:25.394785246 (530h52m25.265868291s) [noise=true , address=145.79.227.10:4403 , piece_id=IZP3UFQYXDPVJQIMBMP6LWWZ6TR52FN755X7VYCFTLREMHOIJN3A, seg_pos={0 0}, upload=465581]: Uploading piece... Dec 27 18:21:25.395302331 (530h52m25.266385376s) [noise=true , address=45.91.116.5:4297 , piece_id=NVFVBNP6PWOJR3AT6ZB6LQWZ2JACMX5YK377GY2RE7GDM5FDPJUA, seg_pos={0 0}, upload=465581]: Uploading piece... Dec 27 18:21:25.396222633 (530h52m25.267305678s) [noise=true , address=45.91.116.8:4046 , piece_id=6JWU3XRTODVGYUZBLZTMJJZFKYNTAHXGTJYQ2R7KXQKUBR7QSVKA, seg_pos={0 0}, upload=465581]: Uploading piece... Dec 27 18:21:25.397111025 (530h52m25.26819407s) [noise=true , address=145.79.227.17:4971 , piece_id=UO6L7APQZ46TBPJ5XOUHI2YL3Y5YZ2VAYFAZHOY7CL5C6BEO5G7Q, seg_pos={0 0}, upload=465581]: Uploading piece... Dec 27 18:21:25.398240415 (530h52m25.26932346s) [noise=true , address=145.79.227.36:4517 , piece_id=RLIQWAGNKI5X6WGHYMN7FXPHWDYEVGWXLFMPN7IEZR4ZRDARFIBQ, seg_pos={0 0}, upload=465581]: Uploading piece... Dec 27 18:21:25.402100956 (530h52m25.273184001s) [noise=true , address=145.79.227.48:4301 , piece_id=KKULR64B2HUQ7P4GSUX2XKQKW4DB3GVTD3PDJUOUPU7TOQ6ZKTJA, seg_pos={0 0}, upload=465581]: Uploading piece... Dec 27 18:21:25.405336728 (530h52m25.276419773s) [noise=true , address=145.79.227.45:4231 , piece_id=UX45A3K3LKYEG4YU7BVZKMJWYVUL47C5MMNQRSI32WE4Y62ILAOQ, seg_pos={0 0}, upload=465581]: Uploading piece... Dec 27 18:21:27.033083485 (530h52m26.90416653s) [upload=465581]: Next returned nil segment Dec 27 18:21:27.033095617 (530h52m26.904178662s) [upload=465581]: Waiting for error group managing segments... Dec 27 18:21:27.422573865 (530h52m27.2936569s) [noise=true , address=145.79.227.11:4732 , piece_id=WVWGLOP2LWAJXKXXMVSIXEXUMPYLU5UPROXM4XFGCDWYVBXTEKOA, seg_pos={0 0}, upload=465581]: Done uploading piece. err: <nil> Dec 27 18:21:27.425425164 (530h52m27.296508199s) [noise=true , address=145.79.227.46:4602 , piece_id=OGGOAL326XC6BS7ULRPAM322APXHYL6FPWWVK32LHVANMFGYJWWQ, seg_pos={0 0}, upload=465581]: Done uploading piece. err: <nil> Dec 27 18:21:27.442445785 (530h52m27.313528821s) [noise=true , address=145.79.227.45:4231 , piece_id=UX45A3K3LKYEG4YU7BVZKMJWYVUL47C5MMNQRSI32WE4Y62ILAOQ, seg_pos={0 0}, upload=465581]: Done uploading piece. err: <nil> Dec 27 18:21:27.449930007 (530h52m27.321013062s) [noise=true , address=145.79.227.17:4971 , piece_id=UO6L7APQZ46TBPJ5XOUHI2YL3Y5YZ2VAYFAZHOY7CL5C6BEO5G7Q, seg_pos={0 0}, upload=465581]: Done uploading piece. err: <nil> Dec 27 18:21:27.450121371 (530h52m27.321204406s) [noise=true , address=145.79.227.10:4403 , piece_id=IZP3UFQYXDPVJQIMBMP6LWWZ6TR52FN755X7VYCFTLREMHOIJN3A, seg_pos={0 0}, upload=465581]: Done uploading piece. err: <nil> Dec 27 18:21:27.454633623 (530h52m27.325716658s) [noise=true , address=145.79.227.48:4301 , piece_id=KKULR64B2HUQ7P4GSUX2XKQKW4DB3GVTD3PDJUOUPU7TOQ6ZKTJA, seg_pos={0 0}, upload=465581]: Done uploading piece. err: <nil> Dec 27 18:21:27.455522034 (530h52m27.326605079s) [noise=true , address=145.79.227.2:4113 , piece_id=B3FPIHAGBGSJN2I6BFIO6S2YPXSIHKF7ZBODNV6FGOI7DQRYTIQA, seg_pos={0 0}, upload=465581]: Done uploading piece. err: <nil> Dec 27 18:21:27.456210736 (530h52m27.327293781s) [noise=true , address=145.79.227.4:4185 , piece_id=BSDWYICA7RMLTXX4URC3O6YC25FHVQ6ZRYL7OS2227KE3DUSWJDQ, seg_pos={0 0}, upload=465581]: Done uploading piece. err: <nil> Dec 27 18:21:27.462096273 (530h52m27.333179308s) [noise=true , address=145.79.227.36:4517 , piece_id=RLIQWAGNKI5X6WGHYMN7FXPHWDYEVGWXLFMPN7IEZR4ZRDARFIBQ, seg_pos={0 0}, upload=465581]: Done uploading piece. err: <nil> Dec 27 18:21:27.509391311 (530h52m27.380474346s) [noise=true , address=145.79.227.16:4910 , piece_id=PKF7JCLQKWL7DBRFMZQOTB2RURKDFJ6NMS7YB3AJYDJ2SC633JOQ, seg_pos={0 0}, upload=465581]: Done uploading piece. err: <nil> Dec 27 18:21:27.640310134 (530h52m27.511393169s) [noise=true , address=45.91.116.6:4974 , piece_id=VDRCSP2C5VIHZR6YLLR7WUGURUZOXC2G7MT56IE43PMVVXDD3HZQ, seg_pos={0 0}, upload=465581]: Done uploading piece. err: <nil> Dec 27 18:21:27.671350691 (530h52m27.542433736s) [noise=true , address=45.91.116.5:4297 , piece_id=NVFVBNP6PWOJR3AT6ZB6LQWZ2JACMX5YK377GY2RE7GDM5FDPJUA, seg_pos={0 0}, upload=465581]: Done uploading piece. err: <nil> Dec 27 18:21:27.671881432 (530h52m27.542964477s) [noise=true , address=45.91.116.13:4156 , piece_id=Y5TLN2KADJ7KUBGC6LFHRWSOSH3E2N2WLWM2O2X6O26YRL7LTDQA, seg_pos={0 0}, upload=465581]: Done uploading piece. err: <nil> Dec 27 18:21:27.671891021 (530h52m27.542974056s) [seg_pos={0 0}, upload=465581]: Segment reached optimal threshold of 13 pieces. Dec 27 18:21:27.672486244 (530h52m27.543569279s) [noise=true , address=45.91.116.4:4939 , piece_id=EMKOBZFOAMID6S27D4UMCJL4JKCH2KGR4KPUKXG4P43BIJIIOGVA, seg_pos={0 0}, upload=465581]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: read tcp 188.166.225.170:39392->45.91.116.4:4939: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 27 18:21:27.672494189 (530h52m27.543577234s) [noise=true , address=45.91.116.4:4939 , piece_id=EMKOBZFOAMID6S27D4UMCJL4JKCH2KGR4KPUKXG4P43BIJIIOGVA, seg_pos={0 0}, upload=465581]: Long tail cancellation due to optimal threshold Dec 27 18:21:27.672941320 (530h52m27.544024365s) [noise=true , address=45.91.116.14:4192 , piece_id=4FEHZOQCMKHKWPHV6DCOTEOP7IYE7LQ22PX2WLFZ47IXQZ3IS36Q, seg_pos={0 0}, upload=465581]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs): protocol: read tcp 188.166.225.170:49760->45.91.116.14:4192: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 27 18:21:27.672949005 (530h52m27.54403204s) [noise=true , address=45.91.116.14:4192 , piece_id=4FEHZOQCMKHKWPHV6DCOTEOP7IYE7LQ22PX2WLFZ47IXQZ3IS36Q, seg_pos={0 0}, upload=465581]: Long tail cancellation due to optimal threshold Dec 27 18:21:27.673176889 (530h52m27.544259924s) [noise=true , address=45.91.116.8:4046 , piece_id=6JWU3XRTODVGYUZBLZTMJJZFKYNTAHXGTJYQ2R7KXQKUBR7QSVKA, seg_pos={0 0}, upload=465581]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H): protocol: expected piece hash; context canceled; EOF Dec 27 18:21:27.673184854 (530h52m27.544267899s) [noise=true , address=45.91.116.8:4046 , piece_id=6JWU3XRTODVGYUZBLZTMJJZFKYNTAHXGTJYQ2R7KXQKUBR7QSVKA, seg_pos={0 0}, upload=465581]: Long tail cancellation due to optimal threshold Dec 27 18:21:27.673203058 (530h52m27.544286093s) [seg_pos={0 0}, upload=465581]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 18:21:27.673307677 (530h52m27.544390713s) [upload=465581]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 18:21:27.673312197 (530h52m27.544395232s) [upload=465581]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 18:21:27.963559300 (530h52m27.834642355s) [upload=465581]: Done uploading segments... Dec 27 18:21:27.963583115 (530h52m27.83466616s) [upload=465581]: Upload finished. err: <nil> Duration: 15.249241113s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [....X.......................................................] duration: 3.583378069s trace: https://jaeger.dev.storj.io/trace/3bddfef0dfa96557 FAIL (2025-12-27 18:25:36 UTC) orbiter warning: context deadline exceeded Duration: 3.583405521s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [....XXX..XX....X...X.X........X.......X..X.XXXXXXX...X......] duration: 6.138962526s trace: https://jaeger.dev.storj.io/trace/4981aef9a433490 FAIL (2025-12-27 18:25:12 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, Piece ID: VDRCSP2C5VIHZR6YLLR7WUGURUZOXC2G7MT56IE43PMVVXDD3HZQ) context deadline exceeded; piecestore: (Node ID: 12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW, Piece ID: NVFVBNP6PWOJR3AT6ZB6LQWZ2JACMX5YK377GY2RE7GDM5FDPJUA) context deadline exceeded ======LOGS====== Dec 27 18:21:25.144885992 (530h52m25.015969028s) [upload=465581]: Starting upload Dec 27 18:21:25.144948482 (530h52m25.016031517s) [upload=465581]: Done starting upload Dec 27 18:21:25.145039054 (530h52m25.016122089s) [upload=465581]: Uploading segments... Dec 27 18:21:25.145258050 (530h52m25.016341095s) [upload=465581]: Got next segment. Inline: false Dec 27 18:21:25.145296154 (530h52m25.016379189s) [upload=465581]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 18:21:25.145301123 (530h52m25.016384158s) [upload=465581]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 18:21:25.389658082 (530h52m25.260741127s) [seg_pos={0 0}, upload=465581]: Begin upload segment... Dec 27 18:21:25.389719219 (530h52m25.260802264s) [seg_pos={0 0}, upload=465581]: Done begin upload segment. Dec 27 18:21:25.389751040 (530h52m25.260834095s) [noise=true , address=145.79.227.16:4910 , piece_id=PKF7JCLQKWL7DBRFMZQOTB2RURKDFJ6NMS7YB3AJYDJ2SC633JOQ, seg_pos={0 0}, upload=465581]: Uploading piece... Dec 27 18:21:25.389827786 (530h52m25.260910821s) [noise=true , address=45.91.116.6:4974 , piece_id=VDRCSP2C5VIHZR6YLLR7WUGURUZOXC2G7MT56IE43PMVVXDD3HZQ, seg_pos={0 0}, upload=465581]: Uploading piece... Dec 27 18:21:25.389925111 (530h52m25.261008146s) [noise=true , address=145.79.227.46:4602 , piece_id=OGGOAL326XC6BS7ULRPAM322APXHYL6FPWWVK32LHVANMFGYJWWQ, seg_pos={0 0}, upload=465581]: Uploading piece... Dec 27 18:21:25.390583174 (530h52m25.261666239s) [noise=true , address=145.79.227.11:4732 , piece_id=WVWGLOP2LWAJXKXXMVSIXEXUMPYLU5UPROXM4XFGCDWYVBXTEKOA, seg_pos={0 0}, upload=465581]: Uploading piece... Dec 27 18:21:25.391844145 (530h52m25.26292719s) [noise=true , address=145.79.227.2:4113 , piece_id=B3FPIHAGBGSJN2I6BFIO6S2YPXSIHKF7ZBODNV6FGOI7DQRYTIQA, seg_pos={0 0}, upload=465581]: Uploading piece... Dec 27 18:21:25.392261891 (530h52m25.263344936s) [noise=true , address=145.79.227.4:4185 , piece_id=BSDWYICA7RMLTXX4URC3O6YC25FHVQ6ZRYL7OS2227KE3DUSWJDQ, seg_pos={0 0}, upload=465581]: Uploading piece... Dec 27 18:21:25.392656372 (530h52m25.263739407s) [noise=true , address=45.91.116.13:4156 , piece_id=Y5TLN2KADJ7KUBGC6LFHRWSOSH3E2N2WLWM2O2X6O26YRL7LTDQA, seg_pos={0 0}, upload=465581]: Uploading piece... Dec 27 18:21:25.393053067 (530h52m25.264136112s) [noise=true , address=45.91.116.4:4939 , piece_id=EMKOBZFOAMID6S27D4UMCJL4JKCH2KGR4KPUKXG4P43BIJIIOGVA, seg_pos={0 0}, upload=465581]: Uploading piece... Dec 27 18:21:25.393516079 (530h52m25.264599114s) [noise=true , address=45.91.116.14:4192 , piece_id=4FEHZOQCMKHKWPHV6DCOTEOP7IYE7LQ22PX2WLFZ47IXQZ3IS36Q, seg_pos={0 0}, upload=465581]: Uploading piece... Dec 27 18:21:25.394785246 (530h52m25.265868291s) [noise=true , address=145.79.227.10:4403 , piece_id=IZP3UFQYXDPVJQIMBMP6LWWZ6TR52FN755X7VYCFTLREMHOIJN3A, seg_pos={0 0}, upload=465581]: Uploading piece... Dec 27 18:21:25.395302331 (530h52m25.266385376s) [noise=true , address=45.91.116.5:4297 , piece_id=NVFVBNP6PWOJR3AT6ZB6LQWZ2JACMX5YK377GY2RE7GDM5FDPJUA, seg_pos={0 0}, upload=465581]: Uploading piece... Dec 27 18:21:25.396222633 (530h52m25.267305678s) [noise=true , address=45.91.116.8:4046 , piece_id=6JWU3XRTODVGYUZBLZTMJJZFKYNTAHXGTJYQ2R7KXQKUBR7QSVKA, seg_pos={0 0}, upload=465581]: Uploading piece... Dec 27 18:21:25.397111025 (530h52m25.26819407s) [noise=true , address=145.79.227.17:4971 , piece_id=UO6L7APQZ46TBPJ5XOUHI2YL3Y5YZ2VAYFAZHOY7CL5C6BEO5G7Q, seg_pos={0 0}, upload=465581]: Uploading piece... Dec 27 18:21:25.398240415 (530h52m25.26932346s) [noise=true , address=145.79.227.36:4517 , piece_id=RLIQWAGNKI5X6WGHYMN7FXPHWDYEVGWXLFMPN7IEZR4ZRDARFIBQ, seg_pos={0 0}, upload=465581]: Uploading piece... Dec 27 18:21:25.402100956 (530h52m25.273184001s) [noise=true , address=145.79.227.48:4301 , piece_id=KKULR64B2HUQ7P4GSUX2XKQKW4DB3GVTD3PDJUOUPU7TOQ6ZKTJA, seg_pos={0 0}, upload=465581]: Uploading piece... Dec 27 18:21:25.405336728 (530h52m25.276419773s) [noise=true , address=145.79.227.45:4231 , piece_id=UX45A3K3LKYEG4YU7BVZKMJWYVUL47C5MMNQRSI32WE4Y62ILAOQ, seg_pos={0 0}, upload=465581]: Uploading piece... Dec 27 18:21:27.033083485 (530h52m26.90416653s) [upload=465581]: Next returned nil segment Dec 27 18:21:27.033095617 (530h52m26.904178662s) [upload=465581]: Waiting for error group managing segments... Dec 27 18:21:27.422573865 (530h52m27.2936569s) [noise=true , address=145.79.227.11:4732 , piece_id=WVWGLOP2LWAJXKXXMVSIXEXUMPYLU5UPROXM4XFGCDWYVBXTEKOA, seg_pos={0 0}, upload=465581]: Done uploading piece. err: <nil> Dec 27 18:21:27.425425164 (530h52m27.296508199s) [noise=true , address=145.79.227.46:4602 , piece_id=OGGOAL326XC6BS7ULRPAM322APXHYL6FPWWVK32LHVANMFGYJWWQ, seg_pos={0 0}, upload=465581]: Done uploading piece. err: <nil> Dec 27 18:21:27.442445785 (530h52m27.313528821s) [noise=true , address=145.79.227.45:4231 , piece_id=UX45A3K3LKYEG4YU7BVZKMJWYVUL47C5MMNQRSI32WE4Y62ILAOQ, seg_pos={0 0}, upload=465581]: Done uploading piece. err: <nil> Dec 27 18:21:27.449930007 (530h52m27.321013062s) [noise=true , address=145.79.227.17:4971 , piece_id=UO6L7APQZ46TBPJ5XOUHI2YL3Y5YZ2VAYFAZHOY7CL5C6BEO5G7Q, seg_pos={0 0}, upload=465581]: Done uploading piece. err: <nil> Dec 27 18:21:27.450121371 (530h52m27.321204406s) [noise=true , address=145.79.227.10:4403 , piece_id=IZP3UFQYXDPVJQIMBMP6LWWZ6TR52FN755X7VYCFTLREMHOIJN3A, seg_pos={0 0}, upload=465581]: Done uploading piece. err: <nil> Dec 27 18:21:27.454633623 (530h52m27.325716658s) [noise=true , address=145.79.227.48:4301 , piece_id=KKULR64B2HUQ7P4GSUX2XKQKW4DB3GVTD3PDJUOUPU7TOQ6ZKTJA, seg_pos={0 0}, upload=465581]: Done uploading piece. err: <nil> Dec 27 18:21:27.455522034 (530h52m27.326605079s) [noise=true , address=145.79.227.2:4113 , piece_id=B3FPIHAGBGSJN2I6BFIO6S2YPXSIHKF7ZBODNV6FGOI7DQRYTIQA, seg_pos={0 0}, upload=465581]: Done uploading piece. err: <nil> Dec 27 18:21:27.456210736 (530h52m27.327293781s) [noise=true , address=145.79.227.4:4185 , piece_id=BSDWYICA7RMLTXX4URC3O6YC25FHVQ6ZRYL7OS2227KE3DUSWJDQ, seg_pos={0 0}, upload=465581]: Done uploading piece. err: <nil> Dec 27 18:21:27.462096273 (530h52m27.333179308s) [noise=true , address=145.79.227.36:4517 , piece_id=RLIQWAGNKI5X6WGHYMN7FXPHWDYEVGWXLFMPN7IEZR4ZRDARFIBQ, seg_pos={0 0}, upload=465581]: Done uploading piece. err: <nil> Dec 27 18:21:27.509391311 (530h52m27.380474346s) [noise=true , address=145.79.227.16:4910 , piece_id=PKF7JCLQKWL7DBRFMZQOTB2RURKDFJ6NMS7YB3AJYDJ2SC633JOQ, seg_pos={0 0}, upload=465581]: Done uploading piece. err: <nil> Dec 27 18:21:27.640310134 (530h52m27.511393169s) [noise=true , address=45.91.116.6:4974 , piece_id=VDRCSP2C5VIHZR6YLLR7WUGURUZOXC2G7MT56IE43PMVVXDD3HZQ, seg_pos={0 0}, upload=465581]: Done uploading piece. err: <nil> Dec 27 18:21:27.671350691 (530h52m27.542433736s) [noise=true , address=45.91.116.5:4297 , piece_id=NVFVBNP6PWOJR3AT6ZB6LQWZ2JACMX5YK377GY2RE7GDM5FDPJUA, seg_pos={0 0}, upload=465581]: Done uploading piece. err: <nil> Dec 27 18:21:27.671881432 (530h52m27.542964477s) [noise=true , address=45.91.116.13:4156 , piece_id=Y5TLN2KADJ7KUBGC6LFHRWSOSH3E2N2WLWM2O2X6O26YRL7LTDQA, seg_pos={0 0}, upload=465581]: Done uploading piece. err: <nil> Dec 27 18:21:27.671891021 (530h52m27.542974056s) [seg_pos={0 0}, upload=465581]: Segment reached optimal threshold of 13 pieces. Dec 27 18:21:27.672486244 (530h52m27.543569279s) [noise=true , address=45.91.116.4:4939 , piece_id=EMKOBZFOAMID6S27D4UMCJL4JKCH2KGR4KPUKXG4P43BIJIIOGVA, seg_pos={0 0}, upload=465581]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: read tcp 188.166.225.170:39392->45.91.116.4:4939: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 27 18:21:27.672494189 (530h52m27.543577234s) [noise=true , address=45.91.116.4:4939 , piece_id=EMKOBZFOAMID6S27D4UMCJL4JKCH2KGR4KPUKXG4P43BIJIIOGVA, seg_pos={0 0}, upload=465581]: Long tail cancellation due to optimal threshold Dec 27 18:21:27.672941320 (530h52m27.544024365s) [noise=true , address=45.91.116.14:4192 , piece_id=4FEHZOQCMKHKWPHV6DCOTEOP7IYE7LQ22PX2WLFZ47IXQZ3IS36Q, seg_pos={0 0}, upload=465581]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs): protocol: read tcp 188.166.225.170:49760->45.91.116.14:4192: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 27 18:21:27.672949005 (530h52m27.54403204s) [noise=true , address=45.91.116.14:4192 , piece_id=4FEHZOQCMKHKWPHV6DCOTEOP7IYE7LQ22PX2WLFZ47IXQZ3IS36Q, seg_pos={0 0}, upload=465581]: Long tail cancellation due to optimal threshold Dec 27 18:21:27.673176889 (530h52m27.544259924s) [noise=true , address=45.91.116.8:4046 , piece_id=6JWU3XRTODVGYUZBLZTMJJZFKYNTAHXGTJYQ2R7KXQKUBR7QSVKA, seg_pos={0 0}, upload=465581]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H): protocol: expected piece hash; context canceled; EOF Dec 27 18:21:27.673184854 (530h52m27.544267899s) [noise=true , address=45.91.116.8:4046 , piece_id=6JWU3XRTODVGYUZBLZTMJJZFKYNTAHXGTJYQ2R7KXQKUBR7QSVKA, seg_pos={0 0}, upload=465581]: Long tail cancellation due to optimal threshold Dec 27 18:21:27.673203058 (530h52m27.544286093s) [seg_pos={0 0}, upload=465581]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 18:21:27.673307677 (530h52m27.544390713s) [upload=465581]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 18:21:27.673312197 (530h52m27.544395232s) [upload=465581]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 18:21:27.963559300 (530h52m27.834642355s) [upload=465581]: Done uploading segments... Dec 27 18:21:27.963583115 (530h52m27.83466616s) [upload=465581]: Upload finished. err: <nil> Duration: 6.138992303s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [X....XXX..XX....X...X.X........X.......X..X.XXXXXXX...X.....] duration: 6.001534146s trace: https://jaeger.dev.storj.io/trace/4981aef9a433490 FAIL (2025-12-27 18:24:35 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, Piece ID: VDRCSP2C5VIHZR6YLLR7WUGURUZOXC2G7MT56IE43PMVVXDD3HZQ) context deadline exceeded; piecestore: (Node ID: 12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW, Piece ID: NVFVBNP6PWOJR3AT6ZB6LQWZ2JACMX5YK377GY2RE7GDM5FDPJUA) context deadline exceeded ======LOGS====== Dec 27 18:21:25.144885992 (530h52m25.015969028s) [upload=465581]: Starting upload Dec 27 18:21:25.144948482 (530h52m25.016031517s) [upload=465581]: Done starting upload Dec 27 18:21:25.145039054 (530h52m25.016122089s) [upload=465581]: Uploading segments... Dec 27 18:21:25.145258050 (530h52m25.016341095s) [upload=465581]: Got next segment. Inline: false Dec 27 18:21:25.145296154 (530h52m25.016379189s) [upload=465581]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 18:21:25.145301123 (530h52m25.016384158s) [upload=465581]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 18:21:25.389658082 (530h52m25.260741127s) [seg_pos={0 0}, upload=465581]: Begin upload segment... Dec 27 18:21:25.389719219 (530h52m25.260802264s) [seg_pos={0 0}, upload=465581]: Done begin upload segment. Dec 27 18:21:25.389751040 (530h52m25.260834095s) [noise=true , address=145.79.227.16:4910 , piece_id=PKF7JCLQKWL7DBRFMZQOTB2RURKDFJ6NMS7YB3AJYDJ2SC633JOQ, seg_pos={0 0}, upload=465581]: Uploading piece... Dec 27 18:21:25.389827786 (530h52m25.260910821s) [noise=true , address=45.91.116.6:4974 , piece_id=VDRCSP2C5VIHZR6YLLR7WUGURUZOXC2G7MT56IE43PMVVXDD3HZQ, seg_pos={0 0}, upload=465581]: Uploading piece... Dec 27 18:21:25.389925111 (530h52m25.261008146s) [noise=true , address=145.79.227.46:4602 , piece_id=OGGOAL326XC6BS7ULRPAM322APXHYL6FPWWVK32LHVANMFGYJWWQ, seg_pos={0 0}, upload=465581]: Uploading piece... Dec 27 18:21:25.390583174 (530h52m25.261666239s) [noise=true , address=145.79.227.11:4732 , piece_id=WVWGLOP2LWAJXKXXMVSIXEXUMPYLU5UPROXM4XFGCDWYVBXTEKOA, seg_pos={0 0}, upload=465581]: Uploading piece... Dec 27 18:21:25.391844145 (530h52m25.26292719s) [noise=true , address=145.79.227.2:4113 , piece_id=B3FPIHAGBGSJN2I6BFIO6S2YPXSIHKF7ZBODNV6FGOI7DQRYTIQA, seg_pos={0 0}, upload=465581]: Uploading piece... Dec 27 18:21:25.392261891 (530h52m25.263344936s) [noise=true , address=145.79.227.4:4185 , piece_id=BSDWYICA7RMLTXX4URC3O6YC25FHVQ6ZRYL7OS2227KE3DUSWJDQ, seg_pos={0 0}, upload=465581]: Uploading piece... Dec 27 18:21:25.392656372 (530h52m25.263739407s) [noise=true , address=45.91.116.13:4156 , piece_id=Y5TLN2KADJ7KUBGC6LFHRWSOSH3E2N2WLWM2O2X6O26YRL7LTDQA, seg_pos={0 0}, upload=465581]: Uploading piece... Dec 27 18:21:25.393053067 (530h52m25.264136112s) [noise=true , address=45.91.116.4:4939 , piece_id=EMKOBZFOAMID6S27D4UMCJL4JKCH2KGR4KPUKXG4P43BIJIIOGVA, seg_pos={0 0}, upload=465581]: Uploading piece... Dec 27 18:21:25.393516079 (530h52m25.264599114s) [noise=true , address=45.91.116.14:4192 , piece_id=4FEHZOQCMKHKWPHV6DCOTEOP7IYE7LQ22PX2WLFZ47IXQZ3IS36Q, seg_pos={0 0}, upload=465581]: Uploading piece... Dec 27 18:21:25.394785246 (530h52m25.265868291s) [noise=true , address=145.79.227.10:4403 , piece_id=IZP3UFQYXDPVJQIMBMP6LWWZ6TR52FN755X7VYCFTLREMHOIJN3A, seg_pos={0 0}, upload=465581]: Uploading piece... Dec 27 18:21:25.395302331 (530h52m25.266385376s) [noise=true , address=45.91.116.5:4297 , piece_id=NVFVBNP6PWOJR3AT6ZB6LQWZ2JACMX5YK377GY2RE7GDM5FDPJUA, seg_pos={0 0}, upload=465581]: Uploading piece... Dec 27 18:21:25.396222633 (530h52m25.267305678s) [noise=true , address=45.91.116.8:4046 , piece_id=6JWU3XRTODVGYUZBLZTMJJZFKYNTAHXGTJYQ2R7KXQKUBR7QSVKA, seg_pos={0 0}, upload=465581]: Uploading piece... Dec 27 18:21:25.397111025 (530h52m25.26819407s) [noise=true , address=145.79.227.17:4971 , piece_id=UO6L7APQZ46TBPJ5XOUHI2YL3Y5YZ2VAYFAZHOY7CL5C6BEO5G7Q, seg_pos={0 0}, upload=465581]: Uploading piece... Dec 27 18:21:25.398240415 (530h52m25.26932346s) [noise=true , address=145.79.227.36:4517 , piece_id=RLIQWAGNKI5X6WGHYMN7FXPHWDYEVGWXLFMPN7IEZR4ZRDARFIBQ, seg_pos={0 0}, upload=465581]: Uploading piece... Dec 27 18:21:25.402100956 (530h52m25.273184001s) [noise=true , address=145.79.227.48:4301 , piece_id=KKULR64B2HUQ7P4GSUX2XKQKW4DB3GVTD3PDJUOUPU7TOQ6ZKTJA, seg_pos={0 0}, upload=465581]: Uploading piece... Dec 27 18:21:25.405336728 (530h52m25.276419773s) [noise=true , address=145.79.227.45:4231 , piece_id=UX45A3K3LKYEG4YU7BVZKMJWYVUL47C5MMNQRSI32WE4Y62ILAOQ, seg_pos={0 0}, upload=465581]: Uploading piece... Dec 27 18:21:27.033083485 (530h52m26.90416653s) [upload=465581]: Next returned nil segment Dec 27 18:21:27.033095617 (530h52m26.904178662s) [upload=465581]: Waiting for error group managing segments... Dec 27 18:21:27.422573865 (530h52m27.2936569s) [noise=true , address=145.79.227.11:4732 , piece_id=WVWGLOP2LWAJXKXXMVSIXEXUMPYLU5UPROXM4XFGCDWYVBXTEKOA, seg_pos={0 0}, upload=465581]: Done uploading piece. err: <nil> Dec 27 18:21:27.425425164 (530h52m27.296508199s) [noise=true , address=145.79.227.46:4602 , piece_id=OGGOAL326XC6BS7ULRPAM322APXHYL6FPWWVK32LHVANMFGYJWWQ, seg_pos={0 0}, upload=465581]: Done uploading piece. err: <nil> Dec 27 18:21:27.442445785 (530h52m27.313528821s) [noise=true , address=145.79.227.45:4231 , piece_id=UX45A3K3LKYEG4YU7BVZKMJWYVUL47C5MMNQRSI32WE4Y62ILAOQ, seg_pos={0 0}, upload=465581]: Done uploading piece. err: <nil> Dec 27 18:21:27.449930007 (530h52m27.321013062s) [noise=true , address=145.79.227.17:4971 , piece_id=UO6L7APQZ46TBPJ5XOUHI2YL3Y5YZ2VAYFAZHOY7CL5C6BEO5G7Q, seg_pos={0 0}, upload=465581]: Done uploading piece. err: <nil> Dec 27 18:21:27.450121371 (530h52m27.321204406s) [noise=true , address=145.79.227.10:4403 , piece_id=IZP3UFQYXDPVJQIMBMP6LWWZ6TR52FN755X7VYCFTLREMHOIJN3A, seg_pos={0 0}, upload=465581]: Done uploading piece. err: <nil> Dec 27 18:21:27.454633623 (530h52m27.325716658s) [noise=true , address=145.79.227.48:4301 , piece_id=KKULR64B2HUQ7P4GSUX2XKQKW4DB3GVTD3PDJUOUPU7TOQ6ZKTJA, seg_pos={0 0}, upload=465581]: Done uploading piece. err: <nil> Dec 27 18:21:27.455522034 (530h52m27.326605079s) [noise=true , address=145.79.227.2:4113 , piece_id=B3FPIHAGBGSJN2I6BFIO6S2YPXSIHKF7ZBODNV6FGOI7DQRYTIQA, seg_pos={0 0}, upload=465581]: Done uploading piece. err: <nil> Dec 27 18:21:27.456210736 (530h52m27.327293781s) [noise=true , address=145.79.227.4:4185 , piece_id=BSDWYICA7RMLTXX4URC3O6YC25FHVQ6ZRYL7OS2227KE3DUSWJDQ, seg_pos={0 0}, upload=465581]: Done uploading piece. err: <nil> Dec 27 18:21:27.462096273 (530h52m27.333179308s) [noise=true , address=145.79.227.36:4517 , piece_id=RLIQWAGNKI5X6WGHYMN7FXPHWDYEVGWXLFMPN7IEZR4ZRDARFIBQ, seg_pos={0 0}, upload=465581]: Done uploading piece. err: <nil> Dec 27 18:21:27.509391311 (530h52m27.380474346s) [noise=true , address=145.79.227.16:4910 , piece_id=PKF7JCLQKWL7DBRFMZQOTB2RURKDFJ6NMS7YB3AJYDJ2SC633JOQ, seg_pos={0 0}, upload=465581]: Done uploading piece. err: <nil> Dec 27 18:21:27.640310134 (530h52m27.511393169s) [noise=true , address=45.91.116.6:4974 , piece_id=VDRCSP2C5VIHZR6YLLR7WUGURUZOXC2G7MT56IE43PMVVXDD3HZQ, seg_pos={0 0}, upload=465581]: Done uploading piece. err: <nil> Dec 27 18:21:27.671350691 (530h52m27.542433736s) [noise=true , address=45.91.116.5:4297 , piece_id=NVFVBNP6PWOJR3AT6ZB6LQWZ2JACMX5YK377GY2RE7GDM5FDPJUA, seg_pos={0 0}, upload=465581]: Done uploading piece. err: <nil> Dec 27 18:21:27.671881432 (530h52m27.542964477s) [noise=true , address=45.91.116.13:4156 , piece_id=Y5TLN2KADJ7KUBGC6LFHRWSOSH3E2N2WLWM2O2X6O26YRL7LTDQA, seg_pos={0 0}, upload=465581]: Done uploading piece. err: <nil> Dec 27 18:21:27.671891021 (530h52m27.542974056s) [seg_pos={0 0}, upload=465581]: Segment reached optimal threshold of 13 pieces. Dec 27 18:21:27.672486244 (530h52m27.543569279s) [noise=true , address=45.91.116.4:4939 , piece_id=EMKOBZFOAMID6S27D4UMCJL4JKCH2KGR4KPUKXG4P43BIJIIOGVA, seg_pos={0 0}, upload=465581]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: read tcp 188.166.225.170:39392->45.91.116.4:4939: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 27 18:21:27.672494189 (530h52m27.543577234s) [noise=true , address=45.91.116.4:4939 , piece_id=EMKOBZFOAMID6S27D4UMCJL4JKCH2KGR4KPUKXG4P43BIJIIOGVA, seg_pos={0 0}, upload=465581]: Long tail cancellation due to optimal threshold Dec 27 18:21:27.672941320 (530h52m27.544024365s) [noise=true , address=45.91.116.14:4192 , piece_id=4FEHZOQCMKHKWPHV6DCOTEOP7IYE7LQ22PX2WLFZ47IXQZ3IS36Q, seg_pos={0 0}, upload=465581]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs): protocol: read tcp 188.166.225.170:49760->45.91.116.14:4192: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 27 18:21:27.672949005 (530h52m27.54403204s) [noise=true , address=45.91.116.14:4192 , piece_id=4FEHZOQCMKHKWPHV6DCOTEOP7IYE7LQ22PX2WLFZ47IXQZ3IS36Q, seg_pos={0 0}, upload=465581]: Long tail cancellation due to optimal threshold Dec 27 18:21:27.673176889 (530h52m27.544259924s) [noise=true , address=45.91.116.8:4046 , piece_id=6JWU3XRTODVGYUZBLZTMJJZFKYNTAHXGTJYQ2R7KXQKUBR7QSVKA, seg_pos={0 0}, upload=465581]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H): protocol: expected piece hash; context canceled; EOF Dec 27 18:21:27.673184854 (530h52m27.544267899s) [noise=true , address=45.91.116.8:4046 , piece_id=6JWU3XRTODVGYUZBLZTMJJZFKYNTAHXGTJYQ2R7KXQKUBR7QSVKA, seg_pos={0 0}, upload=465581]: Long tail cancellation due to optimal threshold Dec 27 18:21:27.673203058 (530h52m27.544286093s) [seg_pos={0 0}, upload=465581]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 18:21:27.673307677 (530h52m27.544390713s) [upload=465581]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 18:21:27.673312197 (530h52m27.544395232s) [upload=465581]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 18:21:27.963559300 (530h52m27.834642355s) [upload=465581]: Done uploading segments... Dec 27 18:21:27.963583115 (530h52m27.83466616s) [upload=465581]: Upload finished. err: <nil> Duration: 6.0015675s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.....X......................................................] duration: 3.505256485s trace: https://jaeger.dev.storj.io/trace/3bddfef0dfa96557 FAIL (2025-12-27 18:24:32 UTC) orbiter warning: context deadline exceeded Duration: 3.505284629s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [.X....XXX..XX....X...X.X........X.......X..X.XXXXXXX...X....] duration: 5.650115742s trace: https://jaeger.dev.storj.io/trace/4981aef9a433490 FAIL (2025-12-27 18:24:17 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, Piece ID: VDRCSP2C5VIHZR6YLLR7WUGURUZOXC2G7MT56IE43PMVVXDD3HZQ) context deadline exceeded; piecestore: (Node ID: 12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW, Piece ID: NVFVBNP6PWOJR3AT6ZB6LQWZ2JACMX5YK377GY2RE7GDM5FDPJUA) context deadline exceeded ======LOGS====== Dec 27 18:21:25.144885992 (530h52m25.015969028s) [upload=465581]: Starting upload Dec 27 18:21:25.144948482 (530h52m25.016031517s) [upload=465581]: Done starting upload Dec 27 18:21:25.145039054 (530h52m25.016122089s) [upload=465581]: Uploading segments... Dec 27 18:21:25.145258050 (530h52m25.016341095s) [upload=465581]: Got next segment. Inline: false Dec 27 18:21:25.145296154 (530h52m25.016379189s) [upload=465581]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 18:21:25.145301123 (530h52m25.016384158s) [upload=465581]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 18:21:25.389658082 (530h52m25.260741127s) [seg_pos={0 0}, upload=465581]: Begin upload segment... Dec 27 18:21:25.389719219 (530h52m25.260802264s) [seg_pos={0 0}, upload=465581]: Done begin upload segment. Dec 27 18:21:25.389751040 (530h52m25.260834095s) [noise=true , address=145.79.227.16:4910 , piece_id=PKF7JCLQKWL7DBRFMZQOTB2RURKDFJ6NMS7YB3AJYDJ2SC633JOQ, seg_pos={0 0}, upload=465581]: Uploading piece... Dec 27 18:21:25.389827786 (530h52m25.260910821s) [noise=true , address=45.91.116.6:4974 , piece_id=VDRCSP2C5VIHZR6YLLR7WUGURUZOXC2G7MT56IE43PMVVXDD3HZQ, seg_pos={0 0}, upload=465581]: Uploading piece... Dec 27 18:21:25.389925111 (530h52m25.261008146s) [noise=true , address=145.79.227.46:4602 , piece_id=OGGOAL326XC6BS7ULRPAM322APXHYL6FPWWVK32LHVANMFGYJWWQ, seg_pos={0 0}, upload=465581]: Uploading piece... Dec 27 18:21:25.390583174 (530h52m25.261666239s) [noise=true , address=145.79.227.11:4732 , piece_id=WVWGLOP2LWAJXKXXMVSIXEXUMPYLU5UPROXM4XFGCDWYVBXTEKOA, seg_pos={0 0}, upload=465581]: Uploading piece... Dec 27 18:21:25.391844145 (530h52m25.26292719s) [noise=true , address=145.79.227.2:4113 , piece_id=B3FPIHAGBGSJN2I6BFIO6S2YPXSIHKF7ZBODNV6FGOI7DQRYTIQA, seg_pos={0 0}, upload=465581]: Uploading piece... Dec 27 18:21:25.392261891 (530h52m25.263344936s) [noise=true , address=145.79.227.4:4185 , piece_id=BSDWYICA7RMLTXX4URC3O6YC25FHVQ6ZRYL7OS2227KE3DUSWJDQ, seg_pos={0 0}, upload=465581]: Uploading piece... Dec 27 18:21:25.392656372 (530h52m25.263739407s) [noise=true , address=45.91.116.13:4156 , piece_id=Y5TLN2KADJ7KUBGC6LFHRWSOSH3E2N2WLWM2O2X6O26YRL7LTDQA, seg_pos={0 0}, upload=465581]: Uploading piece... Dec 27 18:21:25.393053067 (530h52m25.264136112s) [noise=true , address=45.91.116.4:4939 , piece_id=EMKOBZFOAMID6S27D4UMCJL4JKCH2KGR4KPUKXG4P43BIJIIOGVA, seg_pos={0 0}, upload=465581]: Uploading piece... Dec 27 18:21:25.393516079 (530h52m25.264599114s) [noise=true , address=45.91.116.14:4192 , piece_id=4FEHZOQCMKHKWPHV6DCOTEOP7IYE7LQ22PX2WLFZ47IXQZ3IS36Q, seg_pos={0 0}, upload=465581]: Uploading piece... Dec 27 18:21:25.394785246 (530h52m25.265868291s) [noise=true , address=145.79.227.10:4403 , piece_id=IZP3UFQYXDPVJQIMBMP6LWWZ6TR52FN755X7VYCFTLREMHOIJN3A, seg_pos={0 0}, upload=465581]: Uploading piece... Dec 27 18:21:25.395302331 (530h52m25.266385376s) [noise=true , address=45.91.116.5:4297 , piece_id=NVFVBNP6PWOJR3AT6ZB6LQWZ2JACMX5YK377GY2RE7GDM5FDPJUA, seg_pos={0 0}, upload=465581]: Uploading piece... Dec 27 18:21:25.396222633 (530h52m25.267305678s) [noise=true , address=45.91.116.8:4046 , piece_id=6JWU3XRTODVGYUZBLZTMJJZFKYNTAHXGTJYQ2R7KXQKUBR7QSVKA, seg_pos={0 0}, upload=465581]: Uploading piece... Dec 27 18:21:25.397111025 (530h52m25.26819407s) [noise=true , address=145.79.227.17:4971 , piece_id=UO6L7APQZ46TBPJ5XOUHI2YL3Y5YZ2VAYFAZHOY7CL5C6BEO5G7Q, seg_pos={0 0}, upload=465581]: Uploading piece... Dec 27 18:21:25.398240415 (530h52m25.26932346s) [noise=true , address=145.79.227.36:4517 , piece_id=RLIQWAGNKI5X6WGHYMN7FXPHWDYEVGWXLFMPN7IEZR4ZRDARFIBQ, seg_pos={0 0}, upload=465581]: Uploading piece... Dec 27 18:21:25.402100956 (530h52m25.273184001s) [noise=true , address=145.79.227.48:4301 , piece_id=KKULR64B2HUQ7P4GSUX2XKQKW4DB3GVTD3PDJUOUPU7TOQ6ZKTJA, seg_pos={0 0}, upload=465581]: Uploading piece... Dec 27 18:21:25.405336728 (530h52m25.276419773s) [noise=true , address=145.79.227.45:4231 , piece_id=UX45A3K3LKYEG4YU7BVZKMJWYVUL47C5MMNQRSI32WE4Y62ILAOQ, seg_pos={0 0}, upload=465581]: Uploading piece... Dec 27 18:21:27.033083485 (530h52m26.90416653s) [upload=465581]: Next returned nil segment Dec 27 18:21:27.033095617 (530h52m26.904178662s) [upload=465581]: Waiting for error group managing segments... Dec 27 18:21:27.422573865 (530h52m27.2936569s) [noise=true , address=145.79.227.11:4732 , piece_id=WVWGLOP2LWAJXKXXMVSIXEXUMPYLU5UPROXM4XFGCDWYVBXTEKOA, seg_pos={0 0}, upload=465581]: Done uploading piece. err: <nil> Dec 27 18:21:27.425425164 (530h52m27.296508199s) [noise=true , address=145.79.227.46:4602 , piece_id=OGGOAL326XC6BS7ULRPAM322APXHYL6FPWWVK32LHVANMFGYJWWQ, seg_pos={0 0}, upload=465581]: Done uploading piece. err: <nil> Dec 27 18:21:27.442445785 (530h52m27.313528821s) [noise=true , address=145.79.227.45:4231 , piece_id=UX45A3K3LKYEG4YU7BVZKMJWYVUL47C5MMNQRSI32WE4Y62ILAOQ, seg_pos={0 0}, upload=465581]: Done uploading piece. err: <nil> Dec 27 18:21:27.449930007 (530h52m27.321013062s) [noise=true , address=145.79.227.17:4971 , piece_id=UO6L7APQZ46TBPJ5XOUHI2YL3Y5YZ2VAYFAZHOY7CL5C6BEO5G7Q, seg_pos={0 0}, upload=465581]: Done uploading piece. err: <nil> Dec 27 18:21:27.450121371 (530h52m27.321204406s) [noise=true , address=145.79.227.10:4403 , piece_id=IZP3UFQYXDPVJQIMBMP6LWWZ6TR52FN755X7VYCFTLREMHOIJN3A, seg_pos={0 0}, upload=465581]: Done uploading piece. err: <nil> Dec 27 18:21:27.454633623 (530h52m27.325716658s) [noise=true , address=145.79.227.48:4301 , piece_id=KKULR64B2HUQ7P4GSUX2XKQKW4DB3GVTD3PDJUOUPU7TOQ6ZKTJA, seg_pos={0 0}, upload=465581]: Done uploading piece. err: <nil> Dec 27 18:21:27.455522034 (530h52m27.326605079s) [noise=true , address=145.79.227.2:4113 , piece_id=B3FPIHAGBGSJN2I6BFIO6S2YPXSIHKF7ZBODNV6FGOI7DQRYTIQA, seg_pos={0 0}, upload=465581]: Done uploading piece. err: <nil> Dec 27 18:21:27.456210736 (530h52m27.327293781s) [noise=true , address=145.79.227.4:4185 , piece_id=BSDWYICA7RMLTXX4URC3O6YC25FHVQ6ZRYL7OS2227KE3DUSWJDQ, seg_pos={0 0}, upload=465581]: Done uploading piece. err: <nil> Dec 27 18:21:27.462096273 (530h52m27.333179308s) [noise=true , address=145.79.227.36:4517 , piece_id=RLIQWAGNKI5X6WGHYMN7FXPHWDYEVGWXLFMPN7IEZR4ZRDARFIBQ, seg_pos={0 0}, upload=465581]: Done uploading piece. err: <nil> Dec 27 18:21:27.509391311 (530h52m27.380474346s) [noise=true , address=145.79.227.16:4910 , piece_id=PKF7JCLQKWL7DBRFMZQOTB2RURKDFJ6NMS7YB3AJYDJ2SC633JOQ, seg_pos={0 0}, upload=465581]: Done uploading piece. err: <nil> Dec 27 18:21:27.640310134 (530h52m27.511393169s) [noise=true , address=45.91.116.6:4974 , piece_id=VDRCSP2C5VIHZR6YLLR7WUGURUZOXC2G7MT56IE43PMVVXDD3HZQ, seg_pos={0 0}, upload=465581]: Done uploading piece. err: <nil> Dec 27 18:21:27.671350691 (530h52m27.542433736s) [noise=true , address=45.91.116.5:4297 , piece_id=NVFVBNP6PWOJR3AT6ZB6LQWZ2JACMX5YK377GY2RE7GDM5FDPJUA, seg_pos={0 0}, upload=465581]: Done uploading piece. err: <nil> Dec 27 18:21:27.671881432 (530h52m27.542964477s) [noise=true , address=45.91.116.13:4156 , piece_id=Y5TLN2KADJ7KUBGC6LFHRWSOSH3E2N2WLWM2O2X6O26YRL7LTDQA, seg_pos={0 0}, upload=465581]: Done uploading piece. err: <nil> Dec 27 18:21:27.671891021 (530h52m27.542974056s) [seg_pos={0 0}, upload=465581]: Segment reached optimal threshold of 13 pieces. Dec 27 18:21:27.672486244 (530h52m27.543569279s) [noise=true , address=45.91.116.4:4939 , piece_id=EMKOBZFOAMID6S27D4UMCJL4JKCH2KGR4KPUKXG4P43BIJIIOGVA, seg_pos={0 0}, upload=465581]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: read tcp 188.166.225.170:39392->45.91.116.4:4939: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 27 18:21:27.672494189 (530h52m27.543577234s) [noise=true , address=45.91.116.4:4939 , piece_id=EMKOBZFOAMID6S27D4UMCJL4JKCH2KGR4KPUKXG4P43BIJIIOGVA, seg_pos={0 0}, upload=465581]: Long tail cancellation due to optimal threshold Dec 27 18:21:27.672941320 (530h52m27.544024365s) [noise=true , address=45.91.116.14:4192 , piece_id=4FEHZOQCMKHKWPHV6DCOTEOP7IYE7LQ22PX2WLFZ47IXQZ3IS36Q, seg_pos={0 0}, upload=465581]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs): protocol: read tcp 188.166.225.170:49760->45.91.116.14:4192: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 27 18:21:27.672949005 (530h52m27.54403204s) [noise=true , address=45.91.116.14:4192 , piece_id=4FEHZOQCMKHKWPHV6DCOTEOP7IYE7LQ22PX2WLFZ47IXQZ3IS36Q, seg_pos={0 0}, upload=465581]: Long tail cancellation due to optimal threshold Dec 27 18:21:27.673176889 (530h52m27.544259924s) [noise=true , address=45.91.116.8:4046 , piece_id=6JWU3XRTODVGYUZBLZTMJJZFKYNTAHXGTJYQ2R7KXQKUBR7QSVKA, seg_pos={0 0}, upload=465581]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H): protocol: expected piece hash; context canceled; EOF Dec 27 18:21:27.673184854 (530h52m27.544267899s) [noise=true , address=45.91.116.8:4046 , piece_id=6JWU3XRTODVGYUZBLZTMJJZFKYNTAHXGTJYQ2R7KXQKUBR7QSVKA, seg_pos={0 0}, upload=465581]: Long tail cancellation due to optimal threshold Dec 27 18:21:27.673203058 (530h52m27.544286093s) [seg_pos={0 0}, upload=465581]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 18:21:27.673307677 (530h52m27.544390713s) [upload=465581]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 18:21:27.673312197 (530h52m27.544395232s) [upload=465581]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 18:21:27.963559300 (530h52m27.834642355s) [upload=465581]: Done uploading segments... Dec 27 18:21:27.963583115 (530h52m27.83466616s) [upload=465581]: Upload finished. err: <nil> Duration: 5.650146401s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [X.X....XXX..XX....X...X.X........X.......X..X.XXXXXXX...X...] duration: 6.0081466s trace: https://jaeger.dev.storj.io/trace/4981aef9a433490 FAIL (2025-12-27 18:23:33 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, Piece ID: VDRCSP2C5VIHZR6YLLR7WUGURUZOXC2G7MT56IE43PMVVXDD3HZQ) context deadline exceeded; piecestore: (Node ID: 12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW, Piece ID: NVFVBNP6PWOJR3AT6ZB6LQWZ2JACMX5YK377GY2RE7GDM5FDPJUA) context deadline exceeded ======LOGS====== Dec 27 18:21:25.144885992 (530h52m25.015969028s) [upload=465581]: Starting upload Dec 27 18:21:25.144948482 (530h52m25.016031517s) [upload=465581]: Done starting upload Dec 27 18:21:25.145039054 (530h52m25.016122089s) [upload=465581]: Uploading segments... Dec 27 18:21:25.145258050 (530h52m25.016341095s) [upload=465581]: Got next segment. Inline: false Dec 27 18:21:25.145296154 (530h52m25.016379189s) [upload=465581]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 18:21:25.145301123 (530h52m25.016384158s) [upload=465581]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 18:21:25.389658082 (530h52m25.260741127s) [seg_pos={0 0}, upload=465581]: Begin upload segment... Dec 27 18:21:25.389719219 (530h52m25.260802264s) [seg_pos={0 0}, upload=465581]: Done begin upload segment. Dec 27 18:21:25.389751040 (530h52m25.260834095s) [noise=true , address=145.79.227.16:4910 , piece_id=PKF7JCLQKWL7DBRFMZQOTB2RURKDFJ6NMS7YB3AJYDJ2SC633JOQ, seg_pos={0 0}, upload=465581]: Uploading piece... Dec 27 18:21:25.389827786 (530h52m25.260910821s) [noise=true , address=45.91.116.6:4974 , piece_id=VDRCSP2C5VIHZR6YLLR7WUGURUZOXC2G7MT56IE43PMVVXDD3HZQ, seg_pos={0 0}, upload=465581]: Uploading piece... Dec 27 18:21:25.389925111 (530h52m25.261008146s) [noise=true , address=145.79.227.46:4602 , piece_id=OGGOAL326XC6BS7ULRPAM322APXHYL6FPWWVK32LHVANMFGYJWWQ, seg_pos={0 0}, upload=465581]: Uploading piece... Dec 27 18:21:25.390583174 (530h52m25.261666239s) [noise=true , address=145.79.227.11:4732 , piece_id=WVWGLOP2LWAJXKXXMVSIXEXUMPYLU5UPROXM4XFGCDWYVBXTEKOA, seg_pos={0 0}, upload=465581]: Uploading piece... Dec 27 18:21:25.391844145 (530h52m25.26292719s) [noise=true , address=145.79.227.2:4113 , piece_id=B3FPIHAGBGSJN2I6BFIO6S2YPXSIHKF7ZBODNV6FGOI7DQRYTIQA, seg_pos={0 0}, upload=465581]: Uploading piece... Dec 27 18:21:25.392261891 (530h52m25.263344936s) [noise=true , address=145.79.227.4:4185 , piece_id=BSDWYICA7RMLTXX4URC3O6YC25FHVQ6ZRYL7OS2227KE3DUSWJDQ, seg_pos={0 0}, upload=465581]: Uploading piece... Dec 27 18:21:25.392656372 (530h52m25.263739407s) [noise=true , address=45.91.116.13:4156 , piece_id=Y5TLN2KADJ7KUBGC6LFHRWSOSH3E2N2WLWM2O2X6O26YRL7LTDQA, seg_pos={0 0}, upload=465581]: Uploading piece... Dec 27 18:21:25.393053067 (530h52m25.264136112s) [noise=true , address=45.91.116.4:4939 , piece_id=EMKOBZFOAMID6S27D4UMCJL4JKCH2KGR4KPUKXG4P43BIJIIOGVA, seg_pos={0 0}, upload=465581]: Uploading piece... Dec 27 18:21:25.393516079 (530h52m25.264599114s) [noise=true , address=45.91.116.14:4192 , piece_id=4FEHZOQCMKHKWPHV6DCOTEOP7IYE7LQ22PX2WLFZ47IXQZ3IS36Q, seg_pos={0 0}, upload=465581]: Uploading piece... Dec 27 18:21:25.394785246 (530h52m25.265868291s) [noise=true , address=145.79.227.10:4403 , piece_id=IZP3UFQYXDPVJQIMBMP6LWWZ6TR52FN755X7VYCFTLREMHOIJN3A, seg_pos={0 0}, upload=465581]: Uploading piece... Dec 27 18:21:25.395302331 (530h52m25.266385376s) [noise=true , address=45.91.116.5:4297 , piece_id=NVFVBNP6PWOJR3AT6ZB6LQWZ2JACMX5YK377GY2RE7GDM5FDPJUA, seg_pos={0 0}, upload=465581]: Uploading piece... Dec 27 18:21:25.396222633 (530h52m25.267305678s) [noise=true , address=45.91.116.8:4046 , piece_id=6JWU3XRTODVGYUZBLZTMJJZFKYNTAHXGTJYQ2R7KXQKUBR7QSVKA, seg_pos={0 0}, upload=465581]: Uploading piece... Dec 27 18:21:25.397111025 (530h52m25.26819407s) [noise=true , address=145.79.227.17:4971 , piece_id=UO6L7APQZ46TBPJ5XOUHI2YL3Y5YZ2VAYFAZHOY7CL5C6BEO5G7Q, seg_pos={0 0}, upload=465581]: Uploading piece... Dec 27 18:21:25.398240415 (530h52m25.26932346s) [noise=true , address=145.79.227.36:4517 , piece_id=RLIQWAGNKI5X6WGHYMN7FXPHWDYEVGWXLFMPN7IEZR4ZRDARFIBQ, seg_pos={0 0}, upload=465581]: Uploading piece... Dec 27 18:21:25.402100956 (530h52m25.273184001s) [noise=true , address=145.79.227.48:4301 , piece_id=KKULR64B2HUQ7P4GSUX2XKQKW4DB3GVTD3PDJUOUPU7TOQ6ZKTJA, seg_pos={0 0}, upload=465581]: Uploading piece... Dec 27 18:21:25.405336728 (530h52m25.276419773s) [noise=true , address=145.79.227.45:4231 , piece_id=UX45A3K3LKYEG4YU7BVZKMJWYVUL47C5MMNQRSI32WE4Y62ILAOQ, seg_pos={0 0}, upload=465581]: Uploading piece... Dec 27 18:21:27.033083485 (530h52m26.90416653s) [upload=465581]: Next returned nil segment Dec 27 18:21:27.033095617 (530h52m26.904178662s) [upload=465581]: Waiting for error group managing segments... Dec 27 18:21:27.422573865 (530h52m27.2936569s) [noise=true , address=145.79.227.11:4732 , piece_id=WVWGLOP2LWAJXKXXMVSIXEXUMPYLU5UPROXM4XFGCDWYVBXTEKOA, seg_pos={0 0}, upload=465581]: Done uploading piece. err: <nil> Dec 27 18:21:27.425425164 (530h52m27.296508199s) [noise=true , address=145.79.227.46:4602 , piece_id=OGGOAL326XC6BS7ULRPAM322APXHYL6FPWWVK32LHVANMFGYJWWQ, seg_pos={0 0}, upload=465581]: Done uploading piece. err: <nil> Dec 27 18:21:27.442445785 (530h52m27.313528821s) [noise=true , address=145.79.227.45:4231 , piece_id=UX45A3K3LKYEG4YU7BVZKMJWYVUL47C5MMNQRSI32WE4Y62ILAOQ, seg_pos={0 0}, upload=465581]: Done uploading piece. err: <nil> Dec 27 18:21:27.449930007 (530h52m27.321013062s) [noise=true , address=145.79.227.17:4971 , piece_id=UO6L7APQZ46TBPJ5XOUHI2YL3Y5YZ2VAYFAZHOY7CL5C6BEO5G7Q, seg_pos={0 0}, upload=465581]: Done uploading piece. err: <nil> Dec 27 18:21:27.450121371 (530h52m27.321204406s) [noise=true , address=145.79.227.10:4403 , piece_id=IZP3UFQYXDPVJQIMBMP6LWWZ6TR52FN755X7VYCFTLREMHOIJN3A, seg_pos={0 0}, upload=465581]: Done uploading piece. err: <nil> Dec 27 18:21:27.454633623 (530h52m27.325716658s) [noise=true , address=145.79.227.48:4301 , piece_id=KKULR64B2HUQ7P4GSUX2XKQKW4DB3GVTD3PDJUOUPU7TOQ6ZKTJA, seg_pos={0 0}, upload=465581]: Done uploading piece. err: <nil> Dec 27 18:21:27.455522034 (530h52m27.326605079s) [noise=true , address=145.79.227.2:4113 , piece_id=B3FPIHAGBGSJN2I6BFIO6S2YPXSIHKF7ZBODNV6FGOI7DQRYTIQA, seg_pos={0 0}, upload=465581]: Done uploading piece. err: <nil> Dec 27 18:21:27.456210736 (530h52m27.327293781s) [noise=true , address=145.79.227.4:4185 , piece_id=BSDWYICA7RMLTXX4URC3O6YC25FHVQ6ZRYL7OS2227KE3DUSWJDQ, seg_pos={0 0}, upload=465581]: Done uploading piece. err: <nil> Dec 27 18:21:27.462096273 (530h52m27.333179308s) [noise=true , address=145.79.227.36:4517 , piece_id=RLIQWAGNKI5X6WGHYMN7FXPHWDYEVGWXLFMPN7IEZR4ZRDARFIBQ, seg_pos={0 0}, upload=465581]: Done uploading piece. err: <nil> Dec 27 18:21:27.509391311 (530h52m27.380474346s) [noise=true , address=145.79.227.16:4910 , piece_id=PKF7JCLQKWL7DBRFMZQOTB2RURKDFJ6NMS7YB3AJYDJ2SC633JOQ, seg_pos={0 0}, upload=465581]: Done uploading piece. err: <nil> Dec 27 18:21:27.640310134 (530h52m27.511393169s) [noise=true , address=45.91.116.6:4974 , piece_id=VDRCSP2C5VIHZR6YLLR7WUGURUZOXC2G7MT56IE43PMVVXDD3HZQ, seg_pos={0 0}, upload=465581]: Done uploading piece. err: <nil> Dec 27 18:21:27.671350691 (530h52m27.542433736s) [noise=true , address=45.91.116.5:4297 , piece_id=NVFVBNP6PWOJR3AT6ZB6LQWZ2JACMX5YK377GY2RE7GDM5FDPJUA, seg_pos={0 0}, upload=465581]: Done uploading piece. err: <nil> Dec 27 18:21:27.671881432 (530h52m27.542964477s) [noise=true , address=45.91.116.13:4156 , piece_id=Y5TLN2KADJ7KUBGC6LFHRWSOSH3E2N2WLWM2O2X6O26YRL7LTDQA, seg_pos={0 0}, upload=465581]: Done uploading piece. err: <nil> Dec 27 18:21:27.671891021 (530h52m27.542974056s) [seg_pos={0 0}, upload=465581]: Segment reached optimal threshold of 13 pieces. Dec 27 18:21:27.672486244 (530h52m27.543569279s) [noise=true , address=45.91.116.4:4939 , piece_id=EMKOBZFOAMID6S27D4UMCJL4JKCH2KGR4KPUKXG4P43BIJIIOGVA, seg_pos={0 0}, upload=465581]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: read tcp 188.166.225.170:39392->45.91.116.4:4939: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 27 18:21:27.672494189 (530h52m27.543577234s) [noise=true , address=45.91.116.4:4939 , piece_id=EMKOBZFOAMID6S27D4UMCJL4JKCH2KGR4KPUKXG4P43BIJIIOGVA, seg_pos={0 0}, upload=465581]: Long tail cancellation due to optimal threshold Dec 27 18:21:27.672941320 (530h52m27.544024365s) [noise=true , address=45.91.116.14:4192 , piece_id=4FEHZOQCMKHKWPHV6DCOTEOP7IYE7LQ22PX2WLFZ47IXQZ3IS36Q, seg_pos={0 0}, upload=465581]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs): protocol: read tcp 188.166.225.170:49760->45.91.116.14:4192: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 27 18:21:27.672949005 (530h52m27.54403204s) [noise=true , address=45.91.116.14:4192 , piece_id=4FEHZOQCMKHKWPHV6DCOTEOP7IYE7LQ22PX2WLFZ47IXQZ3IS36Q, seg_pos={0 0}, upload=465581]: Long tail cancellation due to optimal threshold Dec 27 18:21:27.673176889 (530h52m27.544259924s) [noise=true , address=45.91.116.8:4046 , piece_id=6JWU3XRTODVGYUZBLZTMJJZFKYNTAHXGTJYQ2R7KXQKUBR7QSVKA, seg_pos={0 0}, upload=465581]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H): protocol: expected piece hash; context canceled; EOF Dec 27 18:21:27.673184854 (530h52m27.544267899s) [noise=true , address=45.91.116.8:4046 , piece_id=6JWU3XRTODVGYUZBLZTMJJZFKYNTAHXGTJYQ2R7KXQKUBR7QSVKA, seg_pos={0 0}, upload=465581]: Long tail cancellation due to optimal threshold Dec 27 18:21:27.673203058 (530h52m27.544286093s) [seg_pos={0 0}, upload=465581]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 18:21:27.673307677 (530h52m27.544390713s) [upload=465581]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 18:21:27.673312197 (530h52m27.544395232s) [upload=465581]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 18:21:27.963559300 (530h52m27.834642355s) [upload=465581]: Done uploading segments... Dec 27 18:21:27.963583115 (530h52m27.83466616s) [upload=465581]: Upload finished. err: <nil> Duration: 6.0081737s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [......X.....................................................] duration: 3.539940973s trace: https://jaeger.dev.storj.io/trace/3bddfef0dfa96557 FAIL (2025-12-27 18:23:24 UTC) orbiter warning: context deadline exceeded Duration: 3.539969597s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [XX.X....XXX..XX....X...X.X........X.......X..X.XXXXXXX...X..] duration: 5.949160817s trace: https://jaeger.dev.storj.io/trace/4981aef9a433490 FAIL (2025-12-27 18:23:17 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, Piece ID: VDRCSP2C5VIHZR6YLLR7WUGURUZOXC2G7MT56IE43PMVVXDD3HZQ) context deadline exceeded; piecestore: (Node ID: 12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW, Piece ID: NVFVBNP6PWOJR3AT6ZB6LQWZ2JACMX5YK377GY2RE7GDM5FDPJUA) context deadline exceeded ======LOGS====== Dec 27 18:21:25.144885992 (530h52m25.015969028s) [upload=465581]: Starting upload Dec 27 18:21:25.144948482 (530h52m25.016031517s) [upload=465581]: Done starting upload Dec 27 18:21:25.145039054 (530h52m25.016122089s) [upload=465581]: Uploading segments... Dec 27 18:21:25.145258050 (530h52m25.016341095s) [upload=465581]: Got next segment. Inline: false Dec 27 18:21:25.145296154 (530h52m25.016379189s) [upload=465581]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 18:21:25.145301123 (530h52m25.016384158s) [upload=465581]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 18:21:25.389658082 (530h52m25.260741127s) [seg_pos={0 0}, upload=465581]: Begin upload segment... Dec 27 18:21:25.389719219 (530h52m25.260802264s) [seg_pos={0 0}, upload=465581]: Done begin upload segment. Dec 27 18:21:25.389751040 (530h52m25.260834095s) [noise=true , address=145.79.227.16:4910 , piece_id=PKF7JCLQKWL7DBRFMZQOTB2RURKDFJ6NMS7YB3AJYDJ2SC633JOQ, seg_pos={0 0}, upload=465581]: Uploading piece... Dec 27 18:21:25.389827786 (530h52m25.260910821s) [noise=true , address=45.91.116.6:4974 , piece_id=VDRCSP2C5VIHZR6YLLR7WUGURUZOXC2G7MT56IE43PMVVXDD3HZQ, seg_pos={0 0}, upload=465581]: Uploading piece... Dec 27 18:21:25.389925111 (530h52m25.261008146s) [noise=true , address=145.79.227.46:4602 , piece_id=OGGOAL326XC6BS7ULRPAM322APXHYL6FPWWVK32LHVANMFGYJWWQ, seg_pos={0 0}, upload=465581]: Uploading piece... Dec 27 18:21:25.390583174 (530h52m25.261666239s) [noise=true , address=145.79.227.11:4732 , piece_id=WVWGLOP2LWAJXKXXMVSIXEXUMPYLU5UPROXM4XFGCDWYVBXTEKOA, seg_pos={0 0}, upload=465581]: Uploading piece... Dec 27 18:21:25.391844145 (530h52m25.26292719s) [noise=true , address=145.79.227.2:4113 , piece_id=B3FPIHAGBGSJN2I6BFIO6S2YPXSIHKF7ZBODNV6FGOI7DQRYTIQA, seg_pos={0 0}, upload=465581]: Uploading piece... Dec 27 18:21:25.392261891 (530h52m25.263344936s) [noise=true , address=145.79.227.4:4185 , piece_id=BSDWYICA7RMLTXX4URC3O6YC25FHVQ6ZRYL7OS2227KE3DUSWJDQ, seg_pos={0 0}, upload=465581]: Uploading piece... Dec 27 18:21:25.392656372 (530h52m25.263739407s) [noise=true , address=45.91.116.13:4156 , piece_id=Y5TLN2KADJ7KUBGC6LFHRWSOSH3E2N2WLWM2O2X6O26YRL7LTDQA, seg_pos={0 0}, upload=465581]: Uploading piece... Dec 27 18:21:25.393053067 (530h52m25.264136112s) [noise=true , address=45.91.116.4:4939 , piece_id=EMKOBZFOAMID6S27D4UMCJL4JKCH2KGR4KPUKXG4P43BIJIIOGVA, seg_pos={0 0}, upload=465581]: Uploading piece... Dec 27 18:21:25.393516079 (530h52m25.264599114s) [noise=true , address=45.91.116.14:4192 , piece_id=4FEHZOQCMKHKWPHV6DCOTEOP7IYE7LQ22PX2WLFZ47IXQZ3IS36Q, seg_pos={0 0}, upload=465581]: Uploading piece... Dec 27 18:21:25.394785246 (530h52m25.265868291s) [noise=true , address=145.79.227.10:4403 , piece_id=IZP3UFQYXDPVJQIMBMP6LWWZ6TR52FN755X7VYCFTLREMHOIJN3A, seg_pos={0 0}, upload=465581]: Uploading piece... Dec 27 18:21:25.395302331 (530h52m25.266385376s) [noise=true , address=45.91.116.5:4297 , piece_id=NVFVBNP6PWOJR3AT6ZB6LQWZ2JACMX5YK377GY2RE7GDM5FDPJUA, seg_pos={0 0}, upload=465581]: Uploading piece... Dec 27 18:21:25.396222633 (530h52m25.267305678s) [noise=true , address=45.91.116.8:4046 , piece_id=6JWU3XRTODVGYUZBLZTMJJZFKYNTAHXGTJYQ2R7KXQKUBR7QSVKA, seg_pos={0 0}, upload=465581]: Uploading piece... Dec 27 18:21:25.397111025 (530h52m25.26819407s) [noise=true , address=145.79.227.17:4971 , piece_id=UO6L7APQZ46TBPJ5XOUHI2YL3Y5YZ2VAYFAZHOY7CL5C6BEO5G7Q, seg_pos={0 0}, upload=465581]: Uploading piece... Dec 27 18:21:25.398240415 (530h52m25.26932346s) [noise=true , address=145.79.227.36:4517 , piece_id=RLIQWAGNKI5X6WGHYMN7FXPHWDYEVGWXLFMPN7IEZR4ZRDARFIBQ, seg_pos={0 0}, upload=465581]: Uploading piece... Dec 27 18:21:25.402100956 (530h52m25.273184001s) [noise=true , address=145.79.227.48:4301 , piece_id=KKULR64B2HUQ7P4GSUX2XKQKW4DB3GVTD3PDJUOUPU7TOQ6ZKTJA, seg_pos={0 0}, upload=465581]: Uploading piece... Dec 27 18:21:25.405336728 (530h52m25.276419773s) [noise=true , address=145.79.227.45:4231 , piece_id=UX45A3K3LKYEG4YU7BVZKMJWYVUL47C5MMNQRSI32WE4Y62ILAOQ, seg_pos={0 0}, upload=465581]: Uploading piece... Dec 27 18:21:27.033083485 (530h52m26.90416653s) [upload=465581]: Next returned nil segment Dec 27 18:21:27.033095617 (530h52m26.904178662s) [upload=465581]: Waiting for error group managing segments... Dec 27 18:21:27.422573865 (530h52m27.2936569s) [noise=true , address=145.79.227.11:4732 , piece_id=WVWGLOP2LWAJXKXXMVSIXEXUMPYLU5UPROXM4XFGCDWYVBXTEKOA, seg_pos={0 0}, upload=465581]: Done uploading piece. err: <nil> Dec 27 18:21:27.425425164 (530h52m27.296508199s) [noise=true , address=145.79.227.46:4602 , piece_id=OGGOAL326XC6BS7ULRPAM322APXHYL6FPWWVK32LHVANMFGYJWWQ, seg_pos={0 0}, upload=465581]: Done uploading piece. err: <nil> Dec 27 18:21:27.442445785 (530h52m27.313528821s) [noise=true , address=145.79.227.45:4231 , piece_id=UX45A3K3LKYEG4YU7BVZKMJWYVUL47C5MMNQRSI32WE4Y62ILAOQ, seg_pos={0 0}, upload=465581]: Done uploading piece. err: <nil> Dec 27 18:21:27.449930007 (530h52m27.321013062s) [noise=true , address=145.79.227.17:4971 , piece_id=UO6L7APQZ46TBPJ5XOUHI2YL3Y5YZ2VAYFAZHOY7CL5C6BEO5G7Q, seg_pos={0 0}, upload=465581]: Done uploading piece. err: <nil> Dec 27 18:21:27.450121371 (530h52m27.321204406s) [noise=true , address=145.79.227.10:4403 , piece_id=IZP3UFQYXDPVJQIMBMP6LWWZ6TR52FN755X7VYCFTLREMHOIJN3A, seg_pos={0 0}, upload=465581]: Done uploading piece. err: <nil> Dec 27 18:21:27.454633623 (530h52m27.325716658s) [noise=true , address=145.79.227.48:4301 , piece_id=KKULR64B2HUQ7P4GSUX2XKQKW4DB3GVTD3PDJUOUPU7TOQ6ZKTJA, seg_pos={0 0}, upload=465581]: Done uploading piece. err: <nil> Dec 27 18:21:27.455522034 (530h52m27.326605079s) [noise=true , address=145.79.227.2:4113 , piece_id=B3FPIHAGBGSJN2I6BFIO6S2YPXSIHKF7ZBODNV6FGOI7DQRYTIQA, seg_pos={0 0}, upload=465581]: Done uploading piece. err: <nil> Dec 27 18:21:27.456210736 (530h52m27.327293781s) [noise=true , address=145.79.227.4:4185 , piece_id=BSDWYICA7RMLTXX4URC3O6YC25FHVQ6ZRYL7OS2227KE3DUSWJDQ, seg_pos={0 0}, upload=465581]: Done uploading piece. err: <nil> Dec 27 18:21:27.462096273 (530h52m27.333179308s) [noise=true , address=145.79.227.36:4517 , piece_id=RLIQWAGNKI5X6WGHYMN7FXPHWDYEVGWXLFMPN7IEZR4ZRDARFIBQ, seg_pos={0 0}, upload=465581]: Done uploading piece. err: <nil> Dec 27 18:21:27.509391311 (530h52m27.380474346s) [noise=true , address=145.79.227.16:4910 , piece_id=PKF7JCLQKWL7DBRFMZQOTB2RURKDFJ6NMS7YB3AJYDJ2SC633JOQ, seg_pos={0 0}, upload=465581]: Done uploading piece. err: <nil> Dec 27 18:21:27.640310134 (530h52m27.511393169s) [noise=true , address=45.91.116.6:4974 , piece_id=VDRCSP2C5VIHZR6YLLR7WUGURUZOXC2G7MT56IE43PMVVXDD3HZQ, seg_pos={0 0}, upload=465581]: Done uploading piece. err: <nil> Dec 27 18:21:27.671350691 (530h52m27.542433736s) [noise=true , address=45.91.116.5:4297 , piece_id=NVFVBNP6PWOJR3AT6ZB6LQWZ2JACMX5YK377GY2RE7GDM5FDPJUA, seg_pos={0 0}, upload=465581]: Done uploading piece. err: <nil> Dec 27 18:21:27.671881432 (530h52m27.542964477s) [noise=true , address=45.91.116.13:4156 , piece_id=Y5TLN2KADJ7KUBGC6LFHRWSOSH3E2N2WLWM2O2X6O26YRL7LTDQA, seg_pos={0 0}, upload=465581]: Done uploading piece. err: <nil> Dec 27 18:21:27.671891021 (530h52m27.542974056s) [seg_pos={0 0}, upload=465581]: Segment reached optimal threshold of 13 pieces. Dec 27 18:21:27.672486244 (530h52m27.543569279s) [noise=true , address=45.91.116.4:4939 , piece_id=EMKOBZFOAMID6S27D4UMCJL4JKCH2KGR4KPUKXG4P43BIJIIOGVA, seg_pos={0 0}, upload=465581]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: read tcp 188.166.225.170:39392->45.91.116.4:4939: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 27 18:21:27.672494189 (530h52m27.543577234s) [noise=true , address=45.91.116.4:4939 , piece_id=EMKOBZFOAMID6S27D4UMCJL4JKCH2KGR4KPUKXG4P43BIJIIOGVA, seg_pos={0 0}, upload=465581]: Long tail cancellation due to optimal threshold Dec 27 18:21:27.672941320 (530h52m27.544024365s) [noise=true , address=45.91.116.14:4192 , piece_id=4FEHZOQCMKHKWPHV6DCOTEOP7IYE7LQ22PX2WLFZ47IXQZ3IS36Q, seg_pos={0 0}, upload=465581]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs): protocol: read tcp 188.166.225.170:49760->45.91.116.14:4192: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 27 18:21:27.672949005 (530h52m27.54403204s) [noise=true , address=45.91.116.14:4192 , piece_id=4FEHZOQCMKHKWPHV6DCOTEOP7IYE7LQ22PX2WLFZ47IXQZ3IS36Q, seg_pos={0 0}, upload=465581]: Long tail cancellation due to optimal threshold Dec 27 18:21:27.673176889 (530h52m27.544259924s) [noise=true , address=45.91.116.8:4046 , piece_id=6JWU3XRTODVGYUZBLZTMJJZFKYNTAHXGTJYQ2R7KXQKUBR7QSVKA, seg_pos={0 0}, upload=465581]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H): protocol: expected piece hash; context canceled; EOF Dec 27 18:21:27.673184854 (530h52m27.544267899s) [noise=true , address=45.91.116.8:4046 , piece_id=6JWU3XRTODVGYUZBLZTMJJZFKYNTAHXGTJYQ2R7KXQKUBR7QSVKA, seg_pos={0 0}, upload=465581]: Long tail cancellation due to optimal threshold Dec 27 18:21:27.673203058 (530h52m27.544286093s) [seg_pos={0 0}, upload=465581]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 18:21:27.673307677 (530h52m27.544390713s) [upload=465581]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 18:21:27.673312197 (530h52m27.544395232s) [upload=465581]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 18:21:27.963559300 (530h52m27.834642355s) [upload=465581]: Done uploading segments... Dec 27 18:21:27.963583115 (530h52m27.83466616s) [upload=465581]: Upload finished. err: <nil> Duration: 5.949187538s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.......X....................................................] duration: 3.587124239s trace: https://jaeger.dev.storj.io/trace/3bddfef0dfa96557 FAIL (2025-12-27 18:22:39 UTC) orbiter warning: context deadline exceeded Duration: 3.587166559s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 23 out of 60 is failed (threshold=3) [XXX.X....XXX..XX....X...X.X........X.......X..X.XXXXXXX...X.] duration: 6.503257439s trace: https://jaeger.dev.storj.io/trace/4981aef9a433490 FAIL (2025-12-27 18:22:31 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, Piece ID: VDRCSP2C5VIHZR6YLLR7WUGURUZOXC2G7MT56IE43PMVVXDD3HZQ) context deadline exceeded; piecestore: (Node ID: 12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW, Piece ID: NVFVBNP6PWOJR3AT6ZB6LQWZ2JACMX5YK377GY2RE7GDM5FDPJUA) context deadline exceeded ======LOGS====== Dec 27 18:21:25.144885992 (530h52m25.015969028s) [upload=465581]: Starting upload Dec 27 18:21:25.144948482 (530h52m25.016031517s) [upload=465581]: Done starting upload Dec 27 18:21:25.145039054 (530h52m25.016122089s) [upload=465581]: Uploading segments... Dec 27 18:21:25.145258050 (530h52m25.016341095s) [upload=465581]: Got next segment. Inline: false Dec 27 18:21:25.145296154 (530h52m25.016379189s) [upload=465581]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 18:21:25.145301123 (530h52m25.016384158s) [upload=465581]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 18:21:25.389658082 (530h52m25.260741127s) [seg_pos={0 0}, upload=465581]: Begin upload segment... Dec 27 18:21:25.389719219 (530h52m25.260802264s) [seg_pos={0 0}, upload=465581]: Done begin upload segment. Dec 27 18:21:25.389751040 (530h52m25.260834095s) [noise=true , address=145.79.227.16:4910 , piece_id=PKF7JCLQKWL7DBRFMZQOTB2RURKDFJ6NMS7YB3AJYDJ2SC633JOQ, seg_pos={0 0}, upload=465581]: Uploading piece... Dec 27 18:21:25.389827786 (530h52m25.260910821s) [noise=true , address=45.91.116.6:4974 , piece_id=VDRCSP2C5VIHZR6YLLR7WUGURUZOXC2G7MT56IE43PMVVXDD3HZQ, seg_pos={0 0}, upload=465581]: Uploading piece... Dec 27 18:21:25.389925111 (530h52m25.261008146s) [noise=true , address=145.79.227.46:4602 , piece_id=OGGOAL326XC6BS7ULRPAM322APXHYL6FPWWVK32LHVANMFGYJWWQ, seg_pos={0 0}, upload=465581]: Uploading piece... Dec 27 18:21:25.390583174 (530h52m25.261666239s) [noise=true , address=145.79.227.11:4732 , piece_id=WVWGLOP2LWAJXKXXMVSIXEXUMPYLU5UPROXM4XFGCDWYVBXTEKOA, seg_pos={0 0}, upload=465581]: Uploading piece... Dec 27 18:21:25.391844145 (530h52m25.26292719s) [noise=true , address=145.79.227.2:4113 , piece_id=B3FPIHAGBGSJN2I6BFIO6S2YPXSIHKF7ZBODNV6FGOI7DQRYTIQA, seg_pos={0 0}, upload=465581]: Uploading piece... Dec 27 18:21:25.392261891 (530h52m25.263344936s) [noise=true , address=145.79.227.4:4185 , piece_id=BSDWYICA7RMLTXX4URC3O6YC25FHVQ6ZRYL7OS2227KE3DUSWJDQ, seg_pos={0 0}, upload=465581]: Uploading piece... Dec 27 18:21:25.392656372 (530h52m25.263739407s) [noise=true , address=45.91.116.13:4156 , piece_id=Y5TLN2KADJ7KUBGC6LFHRWSOSH3E2N2WLWM2O2X6O26YRL7LTDQA, seg_pos={0 0}, upload=465581]: Uploading piece... Dec 27 18:21:25.393053067 (530h52m25.264136112s) [noise=true , address=45.91.116.4:4939 , piece_id=EMKOBZFOAMID6S27D4UMCJL4JKCH2KGR4KPUKXG4P43BIJIIOGVA, seg_pos={0 0}, upload=465581]: Uploading piece... Dec 27 18:21:25.393516079 (530h52m25.264599114s) [noise=true , address=45.91.116.14:4192 , piece_id=4FEHZOQCMKHKWPHV6DCOTEOP7IYE7LQ22PX2WLFZ47IXQZ3IS36Q, seg_pos={0 0}, upload=465581]: Uploading piece... Dec 27 18:21:25.394785246 (530h52m25.265868291s) [noise=true , address=145.79.227.10:4403 , piece_id=IZP3UFQYXDPVJQIMBMP6LWWZ6TR52FN755X7VYCFTLREMHOIJN3A, seg_pos={0 0}, upload=465581]: Uploading piece... Dec 27 18:21:25.395302331 (530h52m25.266385376s) [noise=true , address=45.91.116.5:4297 , piece_id=NVFVBNP6PWOJR3AT6ZB6LQWZ2JACMX5YK377GY2RE7GDM5FDPJUA, seg_pos={0 0}, upload=465581]: Uploading piece... Dec 27 18:21:25.396222633 (530h52m25.267305678s) [noise=true , address=45.91.116.8:4046 , piece_id=6JWU3XRTODVGYUZBLZTMJJZFKYNTAHXGTJYQ2R7KXQKUBR7QSVKA, seg_pos={0 0}, upload=465581]: Uploading piece... Dec 27 18:21:25.397111025 (530h52m25.26819407s) [noise=true , address=145.79.227.17:4971 , piece_id=UO6L7APQZ46TBPJ5XOUHI2YL3Y5YZ2VAYFAZHOY7CL5C6BEO5G7Q, seg_pos={0 0}, upload=465581]: Uploading piece... Dec 27 18:21:25.398240415 (530h52m25.26932346s) [noise=true , address=145.79.227.36:4517 , piece_id=RLIQWAGNKI5X6WGHYMN7FXPHWDYEVGWXLFMPN7IEZR4ZRDARFIBQ, seg_pos={0 0}, upload=465581]: Uploading piece... Dec 27 18:21:25.402100956 (530h52m25.273184001s) [noise=true , address=145.79.227.48:4301 , piece_id=KKULR64B2HUQ7P4GSUX2XKQKW4DB3GVTD3PDJUOUPU7TOQ6ZKTJA, seg_pos={0 0}, upload=465581]: Uploading piece... Dec 27 18:21:25.405336728 (530h52m25.276419773s) [noise=true , address=145.79.227.45:4231 , piece_id=UX45A3K3LKYEG4YU7BVZKMJWYVUL47C5MMNQRSI32WE4Y62ILAOQ, seg_pos={0 0}, upload=465581]: Uploading piece... Dec 27 18:21:27.033083485 (530h52m26.90416653s) [upload=465581]: Next returned nil segment Dec 27 18:21:27.033095617 (530h52m26.904178662s) [upload=465581]: Waiting for error group managing segments... Dec 27 18:21:27.422573865 (530h52m27.2936569s) [noise=true , address=145.79.227.11:4732 , piece_id=WVWGLOP2LWAJXKXXMVSIXEXUMPYLU5UPROXM4XFGCDWYVBXTEKOA, seg_pos={0 0}, upload=465581]: Done uploading piece. err: <nil> Dec 27 18:21:27.425425164 (530h52m27.296508199s) [noise=true , address=145.79.227.46:4602 , piece_id=OGGOAL326XC6BS7ULRPAM322APXHYL6FPWWVK32LHVANMFGYJWWQ, seg_pos={0 0}, upload=465581]: Done uploading piece. err: <nil> Dec 27 18:21:27.442445785 (530h52m27.313528821s) [noise=true , address=145.79.227.45:4231 , piece_id=UX45A3K3LKYEG4YU7BVZKMJWYVUL47C5MMNQRSI32WE4Y62ILAOQ, seg_pos={0 0}, upload=465581]: Done uploading piece. err: <nil> Dec 27 18:21:27.449930007 (530h52m27.321013062s) [noise=true , address=145.79.227.17:4971 , piece_id=UO6L7APQZ46TBPJ5XOUHI2YL3Y5YZ2VAYFAZHOY7CL5C6BEO5G7Q, seg_pos={0 0}, upload=465581]: Done uploading piece. err: <nil> Dec 27 18:21:27.450121371 (530h52m27.321204406s) [noise=true , address=145.79.227.10:4403 , piece_id=IZP3UFQYXDPVJQIMBMP6LWWZ6TR52FN755X7VYCFTLREMHOIJN3A, seg_pos={0 0}, upload=465581]: Done uploading piece. err: <nil> Dec 27 18:21:27.454633623 (530h52m27.325716658s) [noise=true , address=145.79.227.48:4301 , piece_id=KKULR64B2HUQ7P4GSUX2XKQKW4DB3GVTD3PDJUOUPU7TOQ6ZKTJA, seg_pos={0 0}, upload=465581]: Done uploading piece. err: <nil> Dec 27 18:21:27.455522034 (530h52m27.326605079s) [noise=true , address=145.79.227.2:4113 , piece_id=B3FPIHAGBGSJN2I6BFIO6S2YPXSIHKF7ZBODNV6FGOI7DQRYTIQA, seg_pos={0 0}, upload=465581]: Done uploading piece. err: <nil> Dec 27 18:21:27.456210736 (530h52m27.327293781s) [noise=true , address=145.79.227.4:4185 , piece_id=BSDWYICA7RMLTXX4URC3O6YC25FHVQ6ZRYL7OS2227KE3DUSWJDQ, seg_pos={0 0}, upload=465581]: Done uploading piece. err: <nil> Dec 27 18:21:27.462096273 (530h52m27.333179308s) [noise=true , address=145.79.227.36:4517 , piece_id=RLIQWAGNKI5X6WGHYMN7FXPHWDYEVGWXLFMPN7IEZR4ZRDARFIBQ, seg_pos={0 0}, upload=465581]: Done uploading piece. err: <nil> Dec 27 18:21:27.509391311 (530h52m27.380474346s) [noise=true , address=145.79.227.16:4910 , piece_id=PKF7JCLQKWL7DBRFMZQOTB2RURKDFJ6NMS7YB3AJYDJ2SC633JOQ, seg_pos={0 0}, upload=465581]: Done uploading piece. err: <nil> Dec 27 18:21:27.640310134 (530h52m27.511393169s) [noise=true , address=45.91.116.6:4974 , piece_id=VDRCSP2C5VIHZR6YLLR7WUGURUZOXC2G7MT56IE43PMVVXDD3HZQ, seg_pos={0 0}, upload=465581]: Done uploading piece. err: <nil> Dec 27 18:21:27.671350691 (530h52m27.542433736s) [noise=true , address=45.91.116.5:4297 , piece_id=NVFVBNP6PWOJR3AT6ZB6LQWZ2JACMX5YK377GY2RE7GDM5FDPJUA, seg_pos={0 0}, upload=465581]: Done uploading piece. err: <nil> Dec 27 18:21:27.671881432 (530h52m27.542964477s) [noise=true , address=45.91.116.13:4156 , piece_id=Y5TLN2KADJ7KUBGC6LFHRWSOSH3E2N2WLWM2O2X6O26YRL7LTDQA, seg_pos={0 0}, upload=465581]: Done uploading piece. err: <nil> Dec 27 18:21:27.671891021 (530h52m27.542974056s) [seg_pos={0 0}, upload=465581]: Segment reached optimal threshold of 13 pieces. Dec 27 18:21:27.672486244 (530h52m27.543569279s) [noise=true , address=45.91.116.4:4939 , piece_id=EMKOBZFOAMID6S27D4UMCJL4JKCH2KGR4KPUKXG4P43BIJIIOGVA, seg_pos={0 0}, upload=465581]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: read tcp 188.166.225.170:39392->45.91.116.4:4939: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 27 18:21:27.672494189 (530h52m27.543577234s) [noise=true , address=45.91.116.4:4939 , piece_id=EMKOBZFOAMID6S27D4UMCJL4JKCH2KGR4KPUKXG4P43BIJIIOGVA, seg_pos={0 0}, upload=465581]: Long tail cancellation due to optimal threshold Dec 27 18:21:27.672941320 (530h52m27.544024365s) [noise=true , address=45.91.116.14:4192 , piece_id=4FEHZOQCMKHKWPHV6DCOTEOP7IYE7LQ22PX2WLFZ47IXQZ3IS36Q, seg_pos={0 0}, upload=465581]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs): protocol: read tcp 188.166.225.170:49760->45.91.116.14:4192: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 27 18:21:27.672949005 (530h52m27.54403204s) [noise=true , address=45.91.116.14:4192 , piece_id=4FEHZOQCMKHKWPHV6DCOTEOP7IYE7LQ22PX2WLFZ47IXQZ3IS36Q, seg_pos={0 0}, upload=465581]: Long tail cancellation due to optimal threshold Dec 27 18:21:27.673176889 (530h52m27.544259924s) [noise=true , address=45.91.116.8:4046 , piece_id=6JWU3XRTODVGYUZBLZTMJJZFKYNTAHXGTJYQ2R7KXQKUBR7QSVKA, seg_pos={0 0}, upload=465581]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H): protocol: expected piece hash; context canceled; EOF Dec 27 18:21:27.673184854 (530h52m27.544267899s) [noise=true , address=45.91.116.8:4046 , piece_id=6JWU3XRTODVGYUZBLZTMJJZFKYNTAHXGTJYQ2R7KXQKUBR7QSVKA, seg_pos={0 0}, upload=465581]: Long tail cancellation due to optimal threshold Dec 27 18:21:27.673203058 (530h52m27.544286093s) [seg_pos={0 0}, upload=465581]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 18:21:27.673307677 (530h52m27.544390713s) [upload=465581]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 18:21:27.673312197 (530h52m27.544395232s) [upload=465581]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 18:21:27.963559300 (530h52m27.834642355s) [upload=465581]: Done uploading segments... Dec 27 18:21:27.963583115 (530h52m27.83466616s) [upload=465581]: Upload finished. err: <nil> Duration: 6.503288949s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [........X...................................................] duration: 3.588697074s trace: https://jaeger.dev.storj.io/trace/3bddfef0dfa96557 FAIL (2025-12-27 18:22:20 UTC) orbiter warning: context deadline exceeded Duration: 3.588731971s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.........X..................................................] duration: 3.556671963s trace: https://jaeger.dev.storj.io/trace/3bddfef0dfa96557 FAIL (2025-12-27 18:21:56 UTC) orbiter warning: context deadline exceeded Duration: 3.556696328s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 24 out of 60 is failed (threshold=3) [XXXX.X....XXX..XX....X...X.X........X.......X..X.XXXXXXX...X] duration: 30.003264603s trace: https://jaeger.dev.storj.io/trace/4981aef9a433490 FAIL (2025-12-27 18:21:54 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, Piece ID: VDRCSP2C5VIHZR6YLLR7WUGURUZOXC2G7MT56IE43PMVVXDD3HZQ) context deadline exceeded; piecestore: (Node ID: 12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW, Piece ID: NVFVBNP6PWOJR3AT6ZB6LQWZ2JACMX5YK377GY2RE7GDM5FDPJUA) context deadline exceeded ======LOGS====== Dec 27 18:21:25.144885992 (530h52m25.015969028s) [upload=465581]: Starting upload Dec 27 18:21:25.144948482 (530h52m25.016031517s) [upload=465581]: Done starting upload Dec 27 18:21:25.145039054 (530h52m25.016122089s) [upload=465581]: Uploading segments... Dec 27 18:21:25.145258050 (530h52m25.016341095s) [upload=465581]: Got next segment. Inline: false Dec 27 18:21:25.145296154 (530h52m25.016379189s) [upload=465581]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 18:21:25.145301123 (530h52m25.016384158s) [upload=465581]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 18:21:25.389658082 (530h52m25.260741127s) [seg_pos={0 0}, upload=465581]: Begin upload segment... Dec 27 18:21:25.389719219 (530h52m25.260802264s) [seg_pos={0 0}, upload=465581]: Done begin upload segment. Dec 27 18:21:25.389751040 (530h52m25.260834095s) [noise=true , address=145.79.227.16:4910 , piece_id=PKF7JCLQKWL7DBRFMZQOTB2RURKDFJ6NMS7YB3AJYDJ2SC633JOQ, seg_pos={0 0}, upload=465581]: Uploading piece... Dec 27 18:21:25.389827786 (530h52m25.260910821s) [noise=true , address=45.91.116.6:4974 , piece_id=VDRCSP2C5VIHZR6YLLR7WUGURUZOXC2G7MT56IE43PMVVXDD3HZQ, seg_pos={0 0}, upload=465581]: Uploading piece... Dec 27 18:21:25.389925111 (530h52m25.261008146s) [noise=true , address=145.79.227.46:4602 , piece_id=OGGOAL326XC6BS7ULRPAM322APXHYL6FPWWVK32LHVANMFGYJWWQ, seg_pos={0 0}, upload=465581]: Uploading piece... Dec 27 18:21:25.390583174 (530h52m25.261666239s) [noise=true , address=145.79.227.11:4732 , piece_id=WVWGLOP2LWAJXKXXMVSIXEXUMPYLU5UPROXM4XFGCDWYVBXTEKOA, seg_pos={0 0}, upload=465581]: Uploading piece... Dec 27 18:21:25.391844145 (530h52m25.26292719s) [noise=true , address=145.79.227.2:4113 , piece_id=B3FPIHAGBGSJN2I6BFIO6S2YPXSIHKF7ZBODNV6FGOI7DQRYTIQA, seg_pos={0 0}, upload=465581]: Uploading piece... Dec 27 18:21:25.392261891 (530h52m25.263344936s) [noise=true , address=145.79.227.4:4185 , piece_id=BSDWYICA7RMLTXX4URC3O6YC25FHVQ6ZRYL7OS2227KE3DUSWJDQ, seg_pos={0 0}, upload=465581]: Uploading piece... Dec 27 18:21:25.392656372 (530h52m25.263739407s) [noise=true , address=45.91.116.13:4156 , piece_id=Y5TLN2KADJ7KUBGC6LFHRWSOSH3E2N2WLWM2O2X6O26YRL7LTDQA, seg_pos={0 0}, upload=465581]: Uploading piece... Dec 27 18:21:25.393053067 (530h52m25.264136112s) [noise=true , address=45.91.116.4:4939 , piece_id=EMKOBZFOAMID6S27D4UMCJL4JKCH2KGR4KPUKXG4P43BIJIIOGVA, seg_pos={0 0}, upload=465581]: Uploading piece... Dec 27 18:21:25.393516079 (530h52m25.264599114s) [noise=true , address=45.91.116.14:4192 , piece_id=4FEHZOQCMKHKWPHV6DCOTEOP7IYE7LQ22PX2WLFZ47IXQZ3IS36Q, seg_pos={0 0}, upload=465581]: Uploading piece... Dec 27 18:21:25.394785246 (530h52m25.265868291s) [noise=true , address=145.79.227.10:4403 , piece_id=IZP3UFQYXDPVJQIMBMP6LWWZ6TR52FN755X7VYCFTLREMHOIJN3A, seg_pos={0 0}, upload=465581]: Uploading piece... Dec 27 18:21:25.395302331 (530h52m25.266385376s) [noise=true , address=45.91.116.5:4297 , piece_id=NVFVBNP6PWOJR3AT6ZB6LQWZ2JACMX5YK377GY2RE7GDM5FDPJUA, seg_pos={0 0}, upload=465581]: Uploading piece... Dec 27 18:21:25.396222633 (530h52m25.267305678s) [noise=true , address=45.91.116.8:4046 , piece_id=6JWU3XRTODVGYUZBLZTMJJZFKYNTAHXGTJYQ2R7KXQKUBR7QSVKA, seg_pos={0 0}, upload=465581]: Uploading piece... Dec 27 18:21:25.397111025 (530h52m25.26819407s) [noise=true , address=145.79.227.17:4971 , piece_id=UO6L7APQZ46TBPJ5XOUHI2YL3Y5YZ2VAYFAZHOY7CL5C6BEO5G7Q, seg_pos={0 0}, upload=465581]: Uploading piece... Dec 27 18:21:25.398240415 (530h52m25.26932346s) [noise=true , address=145.79.227.36:4517 , piece_id=RLIQWAGNKI5X6WGHYMN7FXPHWDYEVGWXLFMPN7IEZR4ZRDARFIBQ, seg_pos={0 0}, upload=465581]: Uploading piece... Dec 27 18:21:25.402100956 (530h52m25.273184001s) [noise=true , address=145.79.227.48:4301 , piece_id=KKULR64B2HUQ7P4GSUX2XKQKW4DB3GVTD3PDJUOUPU7TOQ6ZKTJA, seg_pos={0 0}, upload=465581]: Uploading piece... Dec 27 18:21:25.405336728 (530h52m25.276419773s) [noise=true , address=145.79.227.45:4231 , piece_id=UX45A3K3LKYEG4YU7BVZKMJWYVUL47C5MMNQRSI32WE4Y62ILAOQ, seg_pos={0 0}, upload=465581]: Uploading piece... Dec 27 18:21:27.033083485 (530h52m26.90416653s) [upload=465581]: Next returned nil segment Dec 27 18:21:27.033095617 (530h52m26.904178662s) [upload=465581]: Waiting for error group managing segments... Dec 27 18:21:27.422573865 (530h52m27.2936569s) [noise=true , address=145.79.227.11:4732 , piece_id=WVWGLOP2LWAJXKXXMVSIXEXUMPYLU5UPROXM4XFGCDWYVBXTEKOA, seg_pos={0 0}, upload=465581]: Done uploading piece. err: <nil> Dec 27 18:21:27.425425164 (530h52m27.296508199s) [noise=true , address=145.79.227.46:4602 , piece_id=OGGOAL326XC6BS7ULRPAM322APXHYL6FPWWVK32LHVANMFGYJWWQ, seg_pos={0 0}, upload=465581]: Done uploading piece. err: <nil> Dec 27 18:21:27.442445785 (530h52m27.313528821s) [noise=true , address=145.79.227.45:4231 , piece_id=UX45A3K3LKYEG4YU7BVZKMJWYVUL47C5MMNQRSI32WE4Y62ILAOQ, seg_pos={0 0}, upload=465581]: Done uploading piece. err: <nil> Dec 27 18:21:27.449930007 (530h52m27.321013062s) [noise=true , address=145.79.227.17:4971 , piece_id=UO6L7APQZ46TBPJ5XOUHI2YL3Y5YZ2VAYFAZHOY7CL5C6BEO5G7Q, seg_pos={0 0}, upload=465581]: Done uploading piece. err: <nil> Dec 27 18:21:27.450121371 (530h52m27.321204406s) [noise=true , address=145.79.227.10:4403 , piece_id=IZP3UFQYXDPVJQIMBMP6LWWZ6TR52FN755X7VYCFTLREMHOIJN3A, seg_pos={0 0}, upload=465581]: Done uploading piece. err: <nil> Dec 27 18:21:27.454633623 (530h52m27.325716658s) [noise=true , address=145.79.227.48:4301 , piece_id=KKULR64B2HUQ7P4GSUX2XKQKW4DB3GVTD3PDJUOUPU7TOQ6ZKTJA, seg_pos={0 0}, upload=465581]: Done uploading piece. err: <nil> Dec 27 18:21:27.455522034 (530h52m27.326605079s) [noise=true , address=145.79.227.2:4113 , piece_id=B3FPIHAGBGSJN2I6BFIO6S2YPXSIHKF7ZBODNV6FGOI7DQRYTIQA, seg_pos={0 0}, upload=465581]: Done uploading piece. err: <nil> Dec 27 18:21:27.456210736 (530h52m27.327293781s) [noise=true , address=145.79.227.4:4185 , piece_id=BSDWYICA7RMLTXX4URC3O6YC25FHVQ6ZRYL7OS2227KE3DUSWJDQ, seg_pos={0 0}, upload=465581]: Done uploading piece. err: <nil> Dec 27 18:21:27.462096273 (530h52m27.333179308s) [noise=true , address=145.79.227.36:4517 , piece_id=RLIQWAGNKI5X6WGHYMN7FXPHWDYEVGWXLFMPN7IEZR4ZRDARFIBQ, seg_pos={0 0}, upload=465581]: Done uploading piece. err: <nil> Dec 27 18:21:27.509391311 (530h52m27.380474346s) [noise=true , address=145.79.227.16:4910 , piece_id=PKF7JCLQKWL7DBRFMZQOTB2RURKDFJ6NMS7YB3AJYDJ2SC633JOQ, seg_pos={0 0}, upload=465581]: Done uploading piece. err: <nil> Dec 27 18:21:27.640310134 (530h52m27.511393169s) [noise=true , address=45.91.116.6:4974 , piece_id=VDRCSP2C5VIHZR6YLLR7WUGURUZOXC2G7MT56IE43PMVVXDD3HZQ, seg_pos={0 0}, upload=465581]: Done uploading piece. err: <nil> Dec 27 18:21:27.671350691 (530h52m27.542433736s) [noise=true , address=45.91.116.5:4297 , piece_id=NVFVBNP6PWOJR3AT6ZB6LQWZ2JACMX5YK377GY2RE7GDM5FDPJUA, seg_pos={0 0}, upload=465581]: Done uploading piece. err: <nil> Dec 27 18:21:27.671881432 (530h52m27.542964477s) [noise=true , address=45.91.116.13:4156 , piece_id=Y5TLN2KADJ7KUBGC6LFHRWSOSH3E2N2WLWM2O2X6O26YRL7LTDQA, seg_pos={0 0}, upload=465581]: Done uploading piece. err: <nil> Dec 27 18:21:27.671891021 (530h52m27.542974056s) [seg_pos={0 0}, upload=465581]: Segment reached optimal threshold of 13 pieces. Dec 27 18:21:27.672486244 (530h52m27.543569279s) [noise=true , address=45.91.116.4:4939 , piece_id=EMKOBZFOAMID6S27D4UMCJL4JKCH2KGR4KPUKXG4P43BIJIIOGVA, seg_pos={0 0}, upload=465581]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: read tcp 188.166.225.170:39392->45.91.116.4:4939: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 27 18:21:27.672494189 (530h52m27.543577234s) [noise=true , address=45.91.116.4:4939 , piece_id=EMKOBZFOAMID6S27D4UMCJL4JKCH2KGR4KPUKXG4P43BIJIIOGVA, seg_pos={0 0}, upload=465581]: Long tail cancellation due to optimal threshold Dec 27 18:21:27.672941320 (530h52m27.544024365s) [noise=true , address=45.91.116.14:4192 , piece_id=4FEHZOQCMKHKWPHV6DCOTEOP7IYE7LQ22PX2WLFZ47IXQZ3IS36Q, seg_pos={0 0}, upload=465581]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs): protocol: read tcp 188.166.225.170:49760->45.91.116.14:4192: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 27 18:21:27.672949005 (530h52m27.54403204s) [noise=true , address=45.91.116.14:4192 , piece_id=4FEHZOQCMKHKWPHV6DCOTEOP7IYE7LQ22PX2WLFZ47IXQZ3IS36Q, seg_pos={0 0}, upload=465581]: Long tail cancellation due to optimal threshold Dec 27 18:21:27.673176889 (530h52m27.544259924s) [noise=true , address=45.91.116.8:4046 , piece_id=6JWU3XRTODVGYUZBLZTMJJZFKYNTAHXGTJYQ2R7KXQKUBR7QSVKA, seg_pos={0 0}, upload=465581]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H): protocol: expected piece hash; context canceled; EOF Dec 27 18:21:27.673184854 (530h52m27.544267899s) [noise=true , address=45.91.116.8:4046 , piece_id=6JWU3XRTODVGYUZBLZTMJJZFKYNTAHXGTJYQ2R7KXQKUBR7QSVKA, seg_pos={0 0}, upload=465581]: Long tail cancellation due to optimal threshold Dec 27 18:21:27.673203058 (530h52m27.544286093s) [seg_pos={0 0}, upload=465581]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 18:21:27.673307677 (530h52m27.544390713s) [upload=465581]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 18:21:27.673312197 (530h52m27.544395232s) [upload=465581]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 18:21:27.963559300 (530h52m27.834642355s) [upload=465581]: Done uploading segments... Dec 27 18:21:27.963583115 (530h52m27.83466616s) [upload=465581]: Upload finished. err: <nil> Duration: 30.0032896s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 23 out of 60 is failed (threshold=3) [.XXXX.X....XXX..XX....X...X.X........X.......X..X.XXXXXXX...] duration: 5.663010263s trace: https://jaeger.dev.storj.io/trace/6aea6e6d013c1516 FAIL (2025-12-27 18:21:02 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, Piece ID: EUF7QLG6CJGERWWATE2N7QGNETUZYWDTFOC6LQVXUFQLKZKZSH5Q) context deadline exceeded; piecestore: (Node ID: 1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, Piece ID: 6KKONXYFBLEKXTJMNQ452QDLMGOFTGOML2W77L7AOCW6IB5D3ELQ) context deadline exceeded; piecestore: (Node ID: 1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t, Piece ID: 57WUY7F7ERDXIXGF5LLR5V5SD3CFEMZHEA4TZY7N2PVHWORCO2GQ) context deadline exceeded ======LOGS====== g segments... Dec 27 18:18:48.727318184 (530h49m48.598401219s) [upload=465546]: Got next segment. Inline: false Dec 27 18:18:48.727380142 (530h49m48.598463187s) [upload=465546]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 18:18:48.727386154 (530h49m48.598469189s) [upload=465546]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 18:18:48.975492044 (530h49m48.846575079s) [seg_pos={0 0}, upload=465546]: Begin upload segment... Dec 27 18:18:48.975558320 (530h49m48.846641355s) [seg_pos={0 0}, upload=465546]: Done begin upload segment. Dec 27 18:18:48.975599548 (530h49m48.846682594s) [noise=true , address=77.90.31.12:4589 , piece_id=3MI7AOPYUTKLWMURIBUKOD7DNP4HYUPST63WDVZXLSEYPGFILXKA, seg_pos={0 0}, upload=465546]: Uploading piece... Dec 27 18:18:48.975677116 (530h49m48.846760171s) [noise=true , address=145.79.227.7:4295 , piece_id=YUTDEZNBRFGICWDWL3UHOYHRIZH7RZSY43AAARQ7H72FVHV2TWRA, seg_pos={0 0}, upload=465546]: Uploading piece... Dec 27 18:18:48.976537595 (530h49m48.84762064s) [noise=true , address=145.79.227.25:4618 , piece_id=657IUHGFLLOLMJVNEWFYZ55FLFQI5QKFO7ENAH6GKYDPI7GQVXFQ, seg_pos={0 0}, upload=465546]: Uploading piece... Dec 27 18:18:48.977260301 (530h49m48.848343346s) [noise=true , address=145.79.227.47:4265 , piece_id=T4WQNGBMENUE32AYQ2MXFAKR3LRYIOVRCJJE37V2XOKOJ4VVIJYA, seg_pos={0 0}, upload=465546]: Uploading piece... Dec 27 18:18:48.977697493 (530h49m48.848780528s) [noise=true , address=145.79.227.3:4151 , piece_id=V5F6ZDZ4S4EUAJI23OZTCWOD5QH5OHOWUENXFHIDNEJXNU6JA7AQ, seg_pos={0 0}, upload=465546]: Uploading piece... Dec 27 18:18:48.977962107 (530h49m48.849045152s) [noise=true , address=77.90.31.10:4519 , piece_id=EUF7QLG6CJGERWWATE2N7QGNETUZYWDTFOC6LQVXUFQLKZKZSH5Q, seg_pos={0 0}, upload=465546]: Uploading piece... Dec 27 18:18:48.978200521 (530h49m48.849283556s) [noise=true , address=45.91.116.7:4010 , piece_id=EY2J4FJ4JPQ2I2EK324SJ2Q43TOYQYPLJWXOUJNCRTCXXOZKMATA, seg_pos={0 0}, upload=465546]: Uploading piece... Dec 27 18:18:48.978629147 (530h49m48.849712182s) [noise=true , address=145.79.227.45:4230 , piece_id=BHEUT4KRORKUD3CFOCIYSYJPISFV3EQMNQJ7Y3FB3KSOGIDUAIJQ, seg_pos={0 0}, upload=465546]: Uploading piece... Dec 27 18:18:48.978701996 (530h49m48.849785031s) [noise=true , address=145.79.227.17:4972 , piece_id=NHQ6L2GHJA7UGARK2BA7HQNSNAFGILDJOCPQBDVWTOGLFXWDSERA, seg_pos={0 0}, upload=465546]: Uploading piece... Dec 27 18:18:48.978951932 (530h49m48.850034967s) [noise=true , address=45.91.116.9:4261 , piece_id=6KKONXYFBLEKXTJMNQ452QDLMGOFTGOML2W77L7AOCW6IB5D3ELQ, seg_pos={0 0}, upload=465546]: Uploading piece... Dec 27 18:18:48.979365720 (530h49m48.850448765s) [noise=true , address=77.90.31.14:4664 , piece_id=J42YQFOCQEVVX32GWSIDMOGZFXZ7KIO5KH5XUJ36XI3KVXYSKPNA, seg_pos={0 0}, upload=465546]: Uploading piece... Dec 27 18:18:48.979489626 (530h49m48.850572681s) [noise=true , address=77.90.31.19:4844 , piece_id=57WUY7F7ERDXIXGF5LLR5V5SD3CFEMZHEA4TZY7N2PVHWORCO2GQ, seg_pos={0 0}, upload=465546]: Uploading piece... Dec 27 18:18:48.979717109 (530h49m48.850800154s) [noise=true , address=77.90.31.13:4627 , piece_id=7IXYPYPEI77UG5VYCM6MGHA5JSEG6PGUUSJMDZ7BKWGB2V24MBSA, seg_pos={0 0}, upload=465546]: Uploading piece... Dec 27 18:18:48.980035897 (530h49m48.851118932s) [noise=true , address=145.79.227.52:4447 , piece_id=KB2TWKDOQFH7JQ24QNMUHWZLIFMO2UEINGNIVSELXYFEY36F5SMA, seg_pos={0 0}, upload=465546]: Uploading piece... Dec 27 18:18:48.980186343 (530h49m48.851269388s) [noise=true , address=45.91.116.4:4939 , piece_id=P2MFR7PJ6GJOENSTD2XRPYMTAUTGNQWNLCTVXLTFNZO3RMP7KDDQ, seg_pos={0 0}, upload=465546]: Uploading piece... Dec 27 18:18:48.981159686 (530h49m48.852242731s) [noise=true , address=145.79.227.37:4566 , piece_id=JBT3U7FWWPQXN7TTXCKP7TVWO5C52RN3PADBCYAN2KHH7S7D7NTA, seg_pos={0 0}, upload=465546]: Uploading piece... Dec 27 18:18:50.593473445 (530h49m50.46455649s) [upload=465546]: Next returned nil segment Dec 27 18:18:50.593485127 (530h49m50.464568162s) [upload=465546]: Waiting for error group managing segments... Dec 27 18:18:50.978438319 (530h49m50.849521364s) [noise=true , address=145.79.227.3:4151 , piece_id=V5F6ZDZ4S4EUAJI23OZTCWOD5QH5OHOWUENXFHIDNEJXNU6JA7AQ, seg_pos={0 0}, upload=465546]: Done uploading piece. err: <nil> Dec 27 18:18:50.983972487 (530h49m50.855055532s) [noise=true , address=145.79.227.37:4566 , piece_id=JBT3U7FWWPQXN7TTXCKP7TVWO5C52RN3PADBCYAN2KHH7S7D7NTA, seg_pos={0 0}, upload=465546]: Done uploading piece. err: <nil> Dec 27 18:18:50.987682160 (530h49m50.858765206s) [noise=true , address=145.79.227.7:4295 , piece_id=YUTDEZNBRFGICWDWL3UHOYHRIZH7RZSY43AAARQ7H72FVHV2TWRA, seg_pos={0 0}, upload=465546]: Done uploading piece. err: <nil> Dec 27 18:18:50.988931730 (530h49m50.860014775s) [noise=true , address=145.79.227.25:4618 , piece_id=657IUHGFLLOLMJVNEWFYZ55FLFQI5QKFO7ENAH6GKYDPI7GQVXFQ, seg_pos={0 0}, upload=465546]: Done uploading piece. err: <nil> Dec 27 18:18:50.991081203 (530h49m50.862164238s) [noise=true , address=145.79.227.52:4447 , piece_id=KB2TWKDOQFH7JQ24QNMUHWZLIFMO2UEINGNIVSELXYFEY36F5SMA, seg_pos={0 0}, upload=465546]: Done uploading piece. err: <nil> Dec 27 18:18:50.999632427 (530h49m50.870715472s) [noise=true , address=145.79.227.45:4230 , piece_id=BHEUT4KRORKUD3CFOCIYSYJPISFV3EQMNQJ7Y3FB3KSOGIDUAIJQ, seg_pos={0 0}, upload=465546]: Done uploading piece. err: <nil> Dec 27 18:18:51.006550861 (530h49m50.877633896s) [noise=true , address=145.79.227.17:4972 , piece_id=NHQ6L2GHJA7UGARK2BA7HQNSNAFGILDJOCPQBDVWTOGLFXWDSERA, seg_pos={0 0}, upload=465546]: Done uploading piece. err: <nil> Dec 27 18:18:51.027665085 (530h49m50.89874813s) [noise=true , address=145.79.227.47:4265 , piece_id=T4WQNGBMENUE32AYQ2MXFAKR3LRYIOVRCJJE37V2XOKOJ4VVIJYA, seg_pos={0 0}, upload=465546]: Done uploading piece. err: <nil> Dec 27 18:18:51.146239203 (530h49m51.017322248s) [noise=true , address=77.90.31.14:4664 , piece_id=J42YQFOCQEVVX32GWSIDMOGZFXZ7KIO5KH5XUJ36XI3KVXYSKPNA, seg_pos={0 0}, upload=465546]: Done uploading piece. err: <nil> Dec 27 18:18:51.164235901 (530h49m51.035318936s) [noise=true , address=77.90.31.10:4519 , piece_id=EUF7QLG6CJGERWWATE2N7QGNETUZYWDTFOC6LQVXUFQLKZKZSH5Q, seg_pos={0 0}, upload=465546]: Done uploading piece. err: <nil> Dec 27 18:18:51.231133206 (530h49m51.102216241s) [noise=true , address=45.91.116.9:4261 , piece_id=6KKONXYFBLEKXTJMNQ452QDLMGOFTGOML2W77L7AOCW6IB5D3ELQ, seg_pos={0 0}, upload=465546]: Done uploading piece. err: <nil> Dec 27 18:18:51.255307578 (530h49m51.126390623s) [noise=true , address=77.90.31.13:4627 , piece_id=7IXYPYPEI77UG5VYCM6MGHA5JSEG6PGUUSJMDZ7BKWGB2V24MBSA, seg_pos={0 0}, upload=465546]: Done uploading piece. err: <nil> Dec 27 18:18:51.282041364 (530h49m51.153124409s) [noise=true , address=77.90.31.19:4844 , piece_id=57WUY7F7ERDXIXGF5LLR5V5SD3CFEMZHEA4TZY7N2PVHWORCO2GQ, seg_pos={0 0}, upload=465546]: Done uploading piece. err: <nil> Dec 27 18:18:51.282068656 (530h49m51.153151701s) [seg_pos={0 0}, upload=465546]: Segment reached optimal threshold of 13 pieces. Dec 27 18:18:51.282286801 (530h49m51.153369847s) [noise=true , address=77.90.31.12:4589 , piece_id=3MI7AOPYUTKLWMURIBUKOD7DNP4HYUPST63WDVZXLSEYPGFILXKA, seg_pos={0 0}, upload=465546]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv): protocol: write tcp 188.166.225.170:39916->77.90.31.12:4589: use of closed network connection; write tcp 188.166.225.170:39916->77.90.31.12:4589: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39916->77.90.31.12:4589: use of closed network connection Dec 27 18:18:51.282294697 (530h49m51.153377742s) [noise=true , address=77.90.31.12:4589 , piece_id=3MI7AOPYUTKLWMURIBUKOD7DNP4HYUPST63WDVZXLSEYPGFILXKA, seg_pos={0 0}, upload=465546]: Long tail cancellation due to optimal threshold Dec 27 18:18:51.282436467 (530h49m51.153519522s) [noise=true , address=45.91.116.4:4939 , piece_id=P2MFR7PJ6GJOENSTD2XRPYMTAUTGNQWNLCTVXLTFNZO3RMP7KDDQ, seg_pos={0 0}, upload=465546]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: write tcp 188.166.225.170:55750->45.91.116.4:4939: use of closed network connection; write tcp 188.166.225.170:55750->45.91.116.4:4939: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55750->45.91.116.4:4939: use of closed network connection Dec 27 18:18:51.282606351 (530h49m51.153689386s) [noise=true , address=45.91.116.4:4939 , piece_id=P2MFR7PJ6GJOENSTD2XRPYMTAUTGNQWNLCTVXLTFNZO3RMP7KDDQ, seg_pos={0 0}, upload=465546]: Long tail cancellation due to optimal threshold Dec 27 18:18:51.282667006 (530h49m51.153750051s) [noise=true , address=45.91.116.7:4010 , piece_id=EY2J4FJ4JPQ2I2EK324SJ2Q43TOYQYPLJWXOUJNCRTCXXOZKMATA, seg_pos={0 0}, upload=465546]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: expected piece hash; context canceled; EOF Dec 27 18:18:51.282676334 (530h49m51.153759379s) [noise=true , address=45.91.116.7:4010 , piece_id=EY2J4FJ4JPQ2I2EK324SJ2Q43TOYQYPLJWXOUJNCRTCXXOZKMATA, seg_pos={0 0}, upload=465546]: Long tail cancellation due to optimal threshold Dec 27 18:18:51.282764662 (530h49m51.153847707s) [seg_pos={0 0}, upload=465546]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 18:18:51.282917964 (530h49m51.154001009s) [upload=465546]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 18:18:51.282924527 (530h49m51.154007572s) [upload=465546]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 18:18:51.598820670 (530h49m51.469903705s) [upload=465546]: Done uploading segments... Dec 27 18:18:51.598840398 (530h49m51.469923433s) [upload=465546]: Upload finished. err: <nil> Duration: 5.663029902s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..........X.................................................] duration: 3.591523516s trace: https://jaeger.dev.storj.io/trace/3bddfef0dfa96557 FAIL (2025-12-27 18:20:57 UTC) orbiter warning: context deadline exceeded Duration: 3.591558984s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 23 out of 60 is failed (threshold=3) [..XXXX.X....XXX..XX....X...X.X........X.......X..X.XXXXXXX..] duration: 26.406633823s trace: https://jaeger.dev.storj.io/trace/6aea6e6d013c1516 FAIL (2025-12-27 18:20:56 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, Piece ID: EUF7QLG6CJGERWWATE2N7QGNETUZYWDTFOC6LQVXUFQLKZKZSH5Q) context deadline exceeded; piecestore: (Node ID: 1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, Piece ID: 6KKONXYFBLEKXTJMNQ452QDLMGOFTGOML2W77L7AOCW6IB5D3ELQ) context deadline exceeded; piecestore: (Node ID: 1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t, Piece ID: 57WUY7F7ERDXIXGF5LLR5V5SD3CFEMZHEA4TZY7N2PVHWORCO2GQ) context deadline exceeded ======LOGS====== g segments... Dec 27 18:18:48.727318184 (530h49m48.598401219s) [upload=465546]: Got next segment. Inline: false Dec 27 18:18:48.727380142 (530h49m48.598463187s) [upload=465546]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 18:18:48.727386154 (530h49m48.598469189s) [upload=465546]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 18:18:48.975492044 (530h49m48.846575079s) [seg_pos={0 0}, upload=465546]: Begin upload segment... Dec 27 18:18:48.975558320 (530h49m48.846641355s) [seg_pos={0 0}, upload=465546]: Done begin upload segment. Dec 27 18:18:48.975599548 (530h49m48.846682594s) [noise=true , address=77.90.31.12:4589 , piece_id=3MI7AOPYUTKLWMURIBUKOD7DNP4HYUPST63WDVZXLSEYPGFILXKA, seg_pos={0 0}, upload=465546]: Uploading piece... Dec 27 18:18:48.975677116 (530h49m48.846760171s) [noise=true , address=145.79.227.7:4295 , piece_id=YUTDEZNBRFGICWDWL3UHOYHRIZH7RZSY43AAARQ7H72FVHV2TWRA, seg_pos={0 0}, upload=465546]: Uploading piece... Dec 27 18:18:48.976537595 (530h49m48.84762064s) [noise=true , address=145.79.227.25:4618 , piece_id=657IUHGFLLOLMJVNEWFYZ55FLFQI5QKFO7ENAH6GKYDPI7GQVXFQ, seg_pos={0 0}, upload=465546]: Uploading piece... Dec 27 18:18:48.977260301 (530h49m48.848343346s) [noise=true , address=145.79.227.47:4265 , piece_id=T4WQNGBMENUE32AYQ2MXFAKR3LRYIOVRCJJE37V2XOKOJ4VVIJYA, seg_pos={0 0}, upload=465546]: Uploading piece... Dec 27 18:18:48.977697493 (530h49m48.848780528s) [noise=true , address=145.79.227.3:4151 , piece_id=V5F6ZDZ4S4EUAJI23OZTCWOD5QH5OHOWUENXFHIDNEJXNU6JA7AQ, seg_pos={0 0}, upload=465546]: Uploading piece... Dec 27 18:18:48.977962107 (530h49m48.849045152s) [noise=true , address=77.90.31.10:4519 , piece_id=EUF7QLG6CJGERWWATE2N7QGNETUZYWDTFOC6LQVXUFQLKZKZSH5Q, seg_pos={0 0}, upload=465546]: Uploading piece... Dec 27 18:18:48.978200521 (530h49m48.849283556s) [noise=true , address=45.91.116.7:4010 , piece_id=EY2J4FJ4JPQ2I2EK324SJ2Q43TOYQYPLJWXOUJNCRTCXXOZKMATA, seg_pos={0 0}, upload=465546]: Uploading piece... Dec 27 18:18:48.978629147 (530h49m48.849712182s) [noise=true , address=145.79.227.45:4230 , piece_id=BHEUT4KRORKUD3CFOCIYSYJPISFV3EQMNQJ7Y3FB3KSOGIDUAIJQ, seg_pos={0 0}, upload=465546]: Uploading piece... Dec 27 18:18:48.978701996 (530h49m48.849785031s) [noise=true , address=145.79.227.17:4972 , piece_id=NHQ6L2GHJA7UGARK2BA7HQNSNAFGILDJOCPQBDVWTOGLFXWDSERA, seg_pos={0 0}, upload=465546]: Uploading piece... Dec 27 18:18:48.978951932 (530h49m48.850034967s) [noise=true , address=45.91.116.9:4261 , piece_id=6KKONXYFBLEKXTJMNQ452QDLMGOFTGOML2W77L7AOCW6IB5D3ELQ, seg_pos={0 0}, upload=465546]: Uploading piece... Dec 27 18:18:48.979365720 (530h49m48.850448765s) [noise=true , address=77.90.31.14:4664 , piece_id=J42YQFOCQEVVX32GWSIDMOGZFXZ7KIO5KH5XUJ36XI3KVXYSKPNA, seg_pos={0 0}, upload=465546]: Uploading piece... Dec 27 18:18:48.979489626 (530h49m48.850572681s) [noise=true , address=77.90.31.19:4844 , piece_id=57WUY7F7ERDXIXGF5LLR5V5SD3CFEMZHEA4TZY7N2PVHWORCO2GQ, seg_pos={0 0}, upload=465546]: Uploading piece... Dec 27 18:18:48.979717109 (530h49m48.850800154s) [noise=true , address=77.90.31.13:4627 , piece_id=7IXYPYPEI77UG5VYCM6MGHA5JSEG6PGUUSJMDZ7BKWGB2V24MBSA, seg_pos={0 0}, upload=465546]: Uploading piece... Dec 27 18:18:48.980035897 (530h49m48.851118932s) [noise=true , address=145.79.227.52:4447 , piece_id=KB2TWKDOQFH7JQ24QNMUHWZLIFMO2UEINGNIVSELXYFEY36F5SMA, seg_pos={0 0}, upload=465546]: Uploading piece... Dec 27 18:18:48.980186343 (530h49m48.851269388s) [noise=true , address=45.91.116.4:4939 , piece_id=P2MFR7PJ6GJOENSTD2XRPYMTAUTGNQWNLCTVXLTFNZO3RMP7KDDQ, seg_pos={0 0}, upload=465546]: Uploading piece... Dec 27 18:18:48.981159686 (530h49m48.852242731s) [noise=true , address=145.79.227.37:4566 , piece_id=JBT3U7FWWPQXN7TTXCKP7TVWO5C52RN3PADBCYAN2KHH7S7D7NTA, seg_pos={0 0}, upload=465546]: Uploading piece... Dec 27 18:18:50.593473445 (530h49m50.46455649s) [upload=465546]: Next returned nil segment Dec 27 18:18:50.593485127 (530h49m50.464568162s) [upload=465546]: Waiting for error group managing segments... Dec 27 18:18:50.978438319 (530h49m50.849521364s) [noise=true , address=145.79.227.3:4151 , piece_id=V5F6ZDZ4S4EUAJI23OZTCWOD5QH5OHOWUENXFHIDNEJXNU6JA7AQ, seg_pos={0 0}, upload=465546]: Done uploading piece. err: <nil> Dec 27 18:18:50.983972487 (530h49m50.855055532s) [noise=true , address=145.79.227.37:4566 , piece_id=JBT3U7FWWPQXN7TTXCKP7TVWO5C52RN3PADBCYAN2KHH7S7D7NTA, seg_pos={0 0}, upload=465546]: Done uploading piece. err: <nil> Dec 27 18:18:50.987682160 (530h49m50.858765206s) [noise=true , address=145.79.227.7:4295 , piece_id=YUTDEZNBRFGICWDWL3UHOYHRIZH7RZSY43AAARQ7H72FVHV2TWRA, seg_pos={0 0}, upload=465546]: Done uploading piece. err: <nil> Dec 27 18:18:50.988931730 (530h49m50.860014775s) [noise=true , address=145.79.227.25:4618 , piece_id=657IUHGFLLOLMJVNEWFYZ55FLFQI5QKFO7ENAH6GKYDPI7GQVXFQ, seg_pos={0 0}, upload=465546]: Done uploading piece. err: <nil> Dec 27 18:18:50.991081203 (530h49m50.862164238s) [noise=true , address=145.79.227.52:4447 , piece_id=KB2TWKDOQFH7JQ24QNMUHWZLIFMO2UEINGNIVSELXYFEY36F5SMA, seg_pos={0 0}, upload=465546]: Done uploading piece. err: <nil> Dec 27 18:18:50.999632427 (530h49m50.870715472s) [noise=true , address=145.79.227.45:4230 , piece_id=BHEUT4KRORKUD3CFOCIYSYJPISFV3EQMNQJ7Y3FB3KSOGIDUAIJQ, seg_pos={0 0}, upload=465546]: Done uploading piece. err: <nil> Dec 27 18:18:51.006550861 (530h49m50.877633896s) [noise=true , address=145.79.227.17:4972 , piece_id=NHQ6L2GHJA7UGARK2BA7HQNSNAFGILDJOCPQBDVWTOGLFXWDSERA, seg_pos={0 0}, upload=465546]: Done uploading piece. err: <nil> Dec 27 18:18:51.027665085 (530h49m50.89874813s) [noise=true , address=145.79.227.47:4265 , piece_id=T4WQNGBMENUE32AYQ2MXFAKR3LRYIOVRCJJE37V2XOKOJ4VVIJYA, seg_pos={0 0}, upload=465546]: Done uploading piece. err: <nil> Dec 27 18:18:51.146239203 (530h49m51.017322248s) [noise=true , address=77.90.31.14:4664 , piece_id=J42YQFOCQEVVX32GWSIDMOGZFXZ7KIO5KH5XUJ36XI3KVXYSKPNA, seg_pos={0 0}, upload=465546]: Done uploading piece. err: <nil> Dec 27 18:18:51.164235901 (530h49m51.035318936s) [noise=true , address=77.90.31.10:4519 , piece_id=EUF7QLG6CJGERWWATE2N7QGNETUZYWDTFOC6LQVXUFQLKZKZSH5Q, seg_pos={0 0}, upload=465546]: Done uploading piece. err: <nil> Dec 27 18:18:51.231133206 (530h49m51.102216241s) [noise=true , address=45.91.116.9:4261 , piece_id=6KKONXYFBLEKXTJMNQ452QDLMGOFTGOML2W77L7AOCW6IB5D3ELQ, seg_pos={0 0}, upload=465546]: Done uploading piece. err: <nil> Dec 27 18:18:51.255307578 (530h49m51.126390623s) [noise=true , address=77.90.31.13:4627 , piece_id=7IXYPYPEI77UG5VYCM6MGHA5JSEG6PGUUSJMDZ7BKWGB2V24MBSA, seg_pos={0 0}, upload=465546]: Done uploading piece. err: <nil> Dec 27 18:18:51.282041364 (530h49m51.153124409s) [noise=true , address=77.90.31.19:4844 , piece_id=57WUY7F7ERDXIXGF5LLR5V5SD3CFEMZHEA4TZY7N2PVHWORCO2GQ, seg_pos={0 0}, upload=465546]: Done uploading piece. err: <nil> Dec 27 18:18:51.282068656 (530h49m51.153151701s) [seg_pos={0 0}, upload=465546]: Segment reached optimal threshold of 13 pieces. Dec 27 18:18:51.282286801 (530h49m51.153369847s) [noise=true , address=77.90.31.12:4589 , piece_id=3MI7AOPYUTKLWMURIBUKOD7DNP4HYUPST63WDVZXLSEYPGFILXKA, seg_pos={0 0}, upload=465546]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv): protocol: write tcp 188.166.225.170:39916->77.90.31.12:4589: use of closed network connection; write tcp 188.166.225.170:39916->77.90.31.12:4589: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39916->77.90.31.12:4589: use of closed network connection Dec 27 18:18:51.282294697 (530h49m51.153377742s) [noise=true , address=77.90.31.12:4589 , piece_id=3MI7AOPYUTKLWMURIBUKOD7DNP4HYUPST63WDVZXLSEYPGFILXKA, seg_pos={0 0}, upload=465546]: Long tail cancellation due to optimal threshold Dec 27 18:18:51.282436467 (530h49m51.153519522s) [noise=true , address=45.91.116.4:4939 , piece_id=P2MFR7PJ6GJOENSTD2XRPYMTAUTGNQWNLCTVXLTFNZO3RMP7KDDQ, seg_pos={0 0}, upload=465546]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: write tcp 188.166.225.170:55750->45.91.116.4:4939: use of closed network connection; write tcp 188.166.225.170:55750->45.91.116.4:4939: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55750->45.91.116.4:4939: use of closed network connection Dec 27 18:18:51.282606351 (530h49m51.153689386s) [noise=true , address=45.91.116.4:4939 , piece_id=P2MFR7PJ6GJOENSTD2XRPYMTAUTGNQWNLCTVXLTFNZO3RMP7KDDQ, seg_pos={0 0}, upload=465546]: Long tail cancellation due to optimal threshold Dec 27 18:18:51.282667006 (530h49m51.153750051s) [noise=true , address=45.91.116.7:4010 , piece_id=EY2J4FJ4JPQ2I2EK324SJ2Q43TOYQYPLJWXOUJNCRTCXXOZKMATA, seg_pos={0 0}, upload=465546]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: expected piece hash; context canceled; EOF Dec 27 18:18:51.282676334 (530h49m51.153759379s) [noise=true , address=45.91.116.7:4010 , piece_id=EY2J4FJ4JPQ2I2EK324SJ2Q43TOYQYPLJWXOUJNCRTCXXOZKMATA, seg_pos={0 0}, upload=465546]: Long tail cancellation due to optimal threshold Dec 27 18:18:51.282764662 (530h49m51.153847707s) [seg_pos={0 0}, upload=465546]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 18:18:51.282917964 (530h49m51.154001009s) [upload=465546]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 18:18:51.282924527 (530h49m51.154007572s) [upload=465546]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 18:18:51.598820670 (530h49m51.469903705s) [upload=465546]: Done uploading segments... Dec 27 18:18:51.598840398 (530h49m51.469923433s) [upload=465546]: Upload finished. err: <nil> Duration: 26.406664221s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...........X................................................] duration: 3.501916165s trace: https://jaeger.dev.storj.io/trace/3bddfef0dfa96557 FAIL (2025-12-27 18:19:54 UTC) orbiter warning: context deadline exceeded Duration: 3.501940321s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 24 out of 60 is failed (threshold=3) [X..XXXX.X....XXX..XX....X...X.X........X.......X..X.XXXXXXX.] duration: 5.955156284s trace: https://jaeger.dev.storj.io/trace/6aea6e6d013c1516 FAIL (2025-12-27 18:19:32 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, Piece ID: EUF7QLG6CJGERWWATE2N7QGNETUZYWDTFOC6LQVXUFQLKZKZSH5Q) context deadline exceeded; piecestore: (Node ID: 1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, Piece ID: 6KKONXYFBLEKXTJMNQ452QDLMGOFTGOML2W77L7AOCW6IB5D3ELQ) context deadline exceeded; piecestore: (Node ID: 1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t, Piece ID: 57WUY7F7ERDXIXGF5LLR5V5SD3CFEMZHEA4TZY7N2PVHWORCO2GQ) context deadline exceeded ======LOGS====== g segments... Dec 27 18:18:48.727318184 (530h49m48.598401219s) [upload=465546]: Got next segment. Inline: false Dec 27 18:18:48.727380142 (530h49m48.598463187s) [upload=465546]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 18:18:48.727386154 (530h49m48.598469189s) [upload=465546]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 18:18:48.975492044 (530h49m48.846575079s) [seg_pos={0 0}, upload=465546]: Begin upload segment... Dec 27 18:18:48.975558320 (530h49m48.846641355s) [seg_pos={0 0}, upload=465546]: Done begin upload segment. Dec 27 18:18:48.975599548 (530h49m48.846682594s) [noise=true , address=77.90.31.12:4589 , piece_id=3MI7AOPYUTKLWMURIBUKOD7DNP4HYUPST63WDVZXLSEYPGFILXKA, seg_pos={0 0}, upload=465546]: Uploading piece... Dec 27 18:18:48.975677116 (530h49m48.846760171s) [noise=true , address=145.79.227.7:4295 , piece_id=YUTDEZNBRFGICWDWL3UHOYHRIZH7RZSY43AAARQ7H72FVHV2TWRA, seg_pos={0 0}, upload=465546]: Uploading piece... Dec 27 18:18:48.976537595 (530h49m48.84762064s) [noise=true , address=145.79.227.25:4618 , piece_id=657IUHGFLLOLMJVNEWFYZ55FLFQI5QKFO7ENAH6GKYDPI7GQVXFQ, seg_pos={0 0}, upload=465546]: Uploading piece... Dec 27 18:18:48.977260301 (530h49m48.848343346s) [noise=true , address=145.79.227.47:4265 , piece_id=T4WQNGBMENUE32AYQ2MXFAKR3LRYIOVRCJJE37V2XOKOJ4VVIJYA, seg_pos={0 0}, upload=465546]: Uploading piece... Dec 27 18:18:48.977697493 (530h49m48.848780528s) [noise=true , address=145.79.227.3:4151 , piece_id=V5F6ZDZ4S4EUAJI23OZTCWOD5QH5OHOWUENXFHIDNEJXNU6JA7AQ, seg_pos={0 0}, upload=465546]: Uploading piece... Dec 27 18:18:48.977962107 (530h49m48.849045152s) [noise=true , address=77.90.31.10:4519 , piece_id=EUF7QLG6CJGERWWATE2N7QGNETUZYWDTFOC6LQVXUFQLKZKZSH5Q, seg_pos={0 0}, upload=465546]: Uploading piece... Dec 27 18:18:48.978200521 (530h49m48.849283556s) [noise=true , address=45.91.116.7:4010 , piece_id=EY2J4FJ4JPQ2I2EK324SJ2Q43TOYQYPLJWXOUJNCRTCXXOZKMATA, seg_pos={0 0}, upload=465546]: Uploading piece... Dec 27 18:18:48.978629147 (530h49m48.849712182s) [noise=true , address=145.79.227.45:4230 , piece_id=BHEUT4KRORKUD3CFOCIYSYJPISFV3EQMNQJ7Y3FB3KSOGIDUAIJQ, seg_pos={0 0}, upload=465546]: Uploading piece... Dec 27 18:18:48.978701996 (530h49m48.849785031s) [noise=true , address=145.79.227.17:4972 , piece_id=NHQ6L2GHJA7UGARK2BA7HQNSNAFGILDJOCPQBDVWTOGLFXWDSERA, seg_pos={0 0}, upload=465546]: Uploading piece... Dec 27 18:18:48.978951932 (530h49m48.850034967s) [noise=true , address=45.91.116.9:4261 , piece_id=6KKONXYFBLEKXTJMNQ452QDLMGOFTGOML2W77L7AOCW6IB5D3ELQ, seg_pos={0 0}, upload=465546]: Uploading piece... Dec 27 18:18:48.979365720 (530h49m48.850448765s) [noise=true , address=77.90.31.14:4664 , piece_id=J42YQFOCQEVVX32GWSIDMOGZFXZ7KIO5KH5XUJ36XI3KVXYSKPNA, seg_pos={0 0}, upload=465546]: Uploading piece... Dec 27 18:18:48.979489626 (530h49m48.850572681s) [noise=true , address=77.90.31.19:4844 , piece_id=57WUY7F7ERDXIXGF5LLR5V5SD3CFEMZHEA4TZY7N2PVHWORCO2GQ, seg_pos={0 0}, upload=465546]: Uploading piece... Dec 27 18:18:48.979717109 (530h49m48.850800154s) [noise=true , address=77.90.31.13:4627 , piece_id=7IXYPYPEI77UG5VYCM6MGHA5JSEG6PGUUSJMDZ7BKWGB2V24MBSA, seg_pos={0 0}, upload=465546]: Uploading piece... Dec 27 18:18:48.980035897 (530h49m48.851118932s) [noise=true , address=145.79.227.52:4447 , piece_id=KB2TWKDOQFH7JQ24QNMUHWZLIFMO2UEINGNIVSELXYFEY36F5SMA, seg_pos={0 0}, upload=465546]: Uploading piece... Dec 27 18:18:48.980186343 (530h49m48.851269388s) [noise=true , address=45.91.116.4:4939 , piece_id=P2MFR7PJ6GJOENSTD2XRPYMTAUTGNQWNLCTVXLTFNZO3RMP7KDDQ, seg_pos={0 0}, upload=465546]: Uploading piece... Dec 27 18:18:48.981159686 (530h49m48.852242731s) [noise=true , address=145.79.227.37:4566 , piece_id=JBT3U7FWWPQXN7TTXCKP7TVWO5C52RN3PADBCYAN2KHH7S7D7NTA, seg_pos={0 0}, upload=465546]: Uploading piece... Dec 27 18:18:50.593473445 (530h49m50.46455649s) [upload=465546]: Next returned nil segment Dec 27 18:18:50.593485127 (530h49m50.464568162s) [upload=465546]: Waiting for error group managing segments... Dec 27 18:18:50.978438319 (530h49m50.849521364s) [noise=true , address=145.79.227.3:4151 , piece_id=V5F6ZDZ4S4EUAJI23OZTCWOD5QH5OHOWUENXFHIDNEJXNU6JA7AQ, seg_pos={0 0}, upload=465546]: Done uploading piece. err: <nil> Dec 27 18:18:50.983972487 (530h49m50.855055532s) [noise=true , address=145.79.227.37:4566 , piece_id=JBT3U7FWWPQXN7TTXCKP7TVWO5C52RN3PADBCYAN2KHH7S7D7NTA, seg_pos={0 0}, upload=465546]: Done uploading piece. err: <nil> Dec 27 18:18:50.987682160 (530h49m50.858765206s) [noise=true , address=145.79.227.7:4295 , piece_id=YUTDEZNBRFGICWDWL3UHOYHRIZH7RZSY43AAARQ7H72FVHV2TWRA, seg_pos={0 0}, upload=465546]: Done uploading piece. err: <nil> Dec 27 18:18:50.988931730 (530h49m50.860014775s) [noise=true , address=145.79.227.25:4618 , piece_id=657IUHGFLLOLMJVNEWFYZ55FLFQI5QKFO7ENAH6GKYDPI7GQVXFQ, seg_pos={0 0}, upload=465546]: Done uploading piece. err: <nil> Dec 27 18:18:50.991081203 (530h49m50.862164238s) [noise=true , address=145.79.227.52:4447 , piece_id=KB2TWKDOQFH7JQ24QNMUHWZLIFMO2UEINGNIVSELXYFEY36F5SMA, seg_pos={0 0}, upload=465546]: Done uploading piece. err: <nil> Dec 27 18:18:50.999632427 (530h49m50.870715472s) [noise=true , address=145.79.227.45:4230 , piece_id=BHEUT4KRORKUD3CFOCIYSYJPISFV3EQMNQJ7Y3FB3KSOGIDUAIJQ, seg_pos={0 0}, upload=465546]: Done uploading piece. err: <nil> Dec 27 18:18:51.006550861 (530h49m50.877633896s) [noise=true , address=145.79.227.17:4972 , piece_id=NHQ6L2GHJA7UGARK2BA7HQNSNAFGILDJOCPQBDVWTOGLFXWDSERA, seg_pos={0 0}, upload=465546]: Done uploading piece. err: <nil> Dec 27 18:18:51.027665085 (530h49m50.89874813s) [noise=true , address=145.79.227.47:4265 , piece_id=T4WQNGBMENUE32AYQ2MXFAKR3LRYIOVRCJJE37V2XOKOJ4VVIJYA, seg_pos={0 0}, upload=465546]: Done uploading piece. err: <nil> Dec 27 18:18:51.146239203 (530h49m51.017322248s) [noise=true , address=77.90.31.14:4664 , piece_id=J42YQFOCQEVVX32GWSIDMOGZFXZ7KIO5KH5XUJ36XI3KVXYSKPNA, seg_pos={0 0}, upload=465546]: Done uploading piece. err: <nil> Dec 27 18:18:51.164235901 (530h49m51.035318936s) [noise=true , address=77.90.31.10:4519 , piece_id=EUF7QLG6CJGERWWATE2N7QGNETUZYWDTFOC6LQVXUFQLKZKZSH5Q, seg_pos={0 0}, upload=465546]: Done uploading piece. err: <nil> Dec 27 18:18:51.231133206 (530h49m51.102216241s) [noise=true , address=45.91.116.9:4261 , piece_id=6KKONXYFBLEKXTJMNQ452QDLMGOFTGOML2W77L7AOCW6IB5D3ELQ, seg_pos={0 0}, upload=465546]: Done uploading piece. err: <nil> Dec 27 18:18:51.255307578 (530h49m51.126390623s) [noise=true , address=77.90.31.13:4627 , piece_id=7IXYPYPEI77UG5VYCM6MGHA5JSEG6PGUUSJMDZ7BKWGB2V24MBSA, seg_pos={0 0}, upload=465546]: Done uploading piece. err: <nil> Dec 27 18:18:51.282041364 (530h49m51.153124409s) [noise=true , address=77.90.31.19:4844 , piece_id=57WUY7F7ERDXIXGF5LLR5V5SD3CFEMZHEA4TZY7N2PVHWORCO2GQ, seg_pos={0 0}, upload=465546]: Done uploading piece. err: <nil> Dec 27 18:18:51.282068656 (530h49m51.153151701s) [seg_pos={0 0}, upload=465546]: Segment reached optimal threshold of 13 pieces. Dec 27 18:18:51.282286801 (530h49m51.153369847s) [noise=true , address=77.90.31.12:4589 , piece_id=3MI7AOPYUTKLWMURIBUKOD7DNP4HYUPST63WDVZXLSEYPGFILXKA, seg_pos={0 0}, upload=465546]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv): protocol: write tcp 188.166.225.170:39916->77.90.31.12:4589: use of closed network connection; write tcp 188.166.225.170:39916->77.90.31.12:4589: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39916->77.90.31.12:4589: use of closed network connection Dec 27 18:18:51.282294697 (530h49m51.153377742s) [noise=true , address=77.90.31.12:4589 , piece_id=3MI7AOPYUTKLWMURIBUKOD7DNP4HYUPST63WDVZXLSEYPGFILXKA, seg_pos={0 0}, upload=465546]: Long tail cancellation due to optimal threshold Dec 27 18:18:51.282436467 (530h49m51.153519522s) [noise=true , address=45.91.116.4:4939 , piece_id=P2MFR7PJ6GJOENSTD2XRPYMTAUTGNQWNLCTVXLTFNZO3RMP7KDDQ, seg_pos={0 0}, upload=465546]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: write tcp 188.166.225.170:55750->45.91.116.4:4939: use of closed network connection; write tcp 188.166.225.170:55750->45.91.116.4:4939: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55750->45.91.116.4:4939: use of closed network connection Dec 27 18:18:51.282606351 (530h49m51.153689386s) [noise=true , address=45.91.116.4:4939 , piece_id=P2MFR7PJ6GJOENSTD2XRPYMTAUTGNQWNLCTVXLTFNZO3RMP7KDDQ, seg_pos={0 0}, upload=465546]: Long tail cancellation due to optimal threshold Dec 27 18:18:51.282667006 (530h49m51.153750051s) [noise=true , address=45.91.116.7:4010 , piece_id=EY2J4FJ4JPQ2I2EK324SJ2Q43TOYQYPLJWXOUJNCRTCXXOZKMATA, seg_pos={0 0}, upload=465546]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: expected piece hash; context canceled; EOF Dec 27 18:18:51.282676334 (530h49m51.153759379s) [noise=true , address=45.91.116.7:4010 , piece_id=EY2J4FJ4JPQ2I2EK324SJ2Q43TOYQYPLJWXOUJNCRTCXXOZKMATA, seg_pos={0 0}, upload=465546]: Long tail cancellation due to optimal threshold Dec 27 18:18:51.282764662 (530h49m51.153847707s) [seg_pos={0 0}, upload=465546]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 18:18:51.282917964 (530h49m51.154001009s) [upload=465546]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 18:18:51.282924527 (530h49m51.154007572s) [upload=465546]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 18:18:51.598820670 (530h49m51.469903705s) [upload=465546]: Done uploading segments... Dec 27 18:18:51.598840398 (530h49m51.469923433s) [upload=465546]: Upload finished. err: <nil> Duration: 5.955187164s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 24 out of 60 is failed (threshold=3) [.X..XXXX.X....XXX..XX....X...X.X........X.......X..X.XXXXXXX] duration: 30.002591872s trace: https://jaeger.dev.storj.io/trace/6aea6e6d013c1516 FAIL (2025-12-27 18:19:18 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, Piece ID: EUF7QLG6CJGERWWATE2N7QGNETUZYWDTFOC6LQVXUFQLKZKZSH5Q) context deadline exceeded; piecestore: (Node ID: 1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, Piece ID: 6KKONXYFBLEKXTJMNQ452QDLMGOFTGOML2W77L7AOCW6IB5D3ELQ) context deadline exceeded; piecestore: (Node ID: 1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t, Piece ID: 57WUY7F7ERDXIXGF5LLR5V5SD3CFEMZHEA4TZY7N2PVHWORCO2GQ) context deadline exceeded ======LOGS====== g segments... Dec 27 18:18:48.727318184 (530h49m48.598401219s) [upload=465546]: Got next segment. Inline: false Dec 27 18:18:48.727380142 (530h49m48.598463187s) [upload=465546]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 18:18:48.727386154 (530h49m48.598469189s) [upload=465546]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 18:18:48.975492044 (530h49m48.846575079s) [seg_pos={0 0}, upload=465546]: Begin upload segment... Dec 27 18:18:48.975558320 (530h49m48.846641355s) [seg_pos={0 0}, upload=465546]: Done begin upload segment. Dec 27 18:18:48.975599548 (530h49m48.846682594s) [noise=true , address=77.90.31.12:4589 , piece_id=3MI7AOPYUTKLWMURIBUKOD7DNP4HYUPST63WDVZXLSEYPGFILXKA, seg_pos={0 0}, upload=465546]: Uploading piece... Dec 27 18:18:48.975677116 (530h49m48.846760171s) [noise=true , address=145.79.227.7:4295 , piece_id=YUTDEZNBRFGICWDWL3UHOYHRIZH7RZSY43AAARQ7H72FVHV2TWRA, seg_pos={0 0}, upload=465546]: Uploading piece... Dec 27 18:18:48.976537595 (530h49m48.84762064s) [noise=true , address=145.79.227.25:4618 , piece_id=657IUHGFLLOLMJVNEWFYZ55FLFQI5QKFO7ENAH6GKYDPI7GQVXFQ, seg_pos={0 0}, upload=465546]: Uploading piece... Dec 27 18:18:48.977260301 (530h49m48.848343346s) [noise=true , address=145.79.227.47:4265 , piece_id=T4WQNGBMENUE32AYQ2MXFAKR3LRYIOVRCJJE37V2XOKOJ4VVIJYA, seg_pos={0 0}, upload=465546]: Uploading piece... Dec 27 18:18:48.977697493 (530h49m48.848780528s) [noise=true , address=145.79.227.3:4151 , piece_id=V5F6ZDZ4S4EUAJI23OZTCWOD5QH5OHOWUENXFHIDNEJXNU6JA7AQ, seg_pos={0 0}, upload=465546]: Uploading piece... Dec 27 18:18:48.977962107 (530h49m48.849045152s) [noise=true , address=77.90.31.10:4519 , piece_id=EUF7QLG6CJGERWWATE2N7QGNETUZYWDTFOC6LQVXUFQLKZKZSH5Q, seg_pos={0 0}, upload=465546]: Uploading piece... Dec 27 18:18:48.978200521 (530h49m48.849283556s) [noise=true , address=45.91.116.7:4010 , piece_id=EY2J4FJ4JPQ2I2EK324SJ2Q43TOYQYPLJWXOUJNCRTCXXOZKMATA, seg_pos={0 0}, upload=465546]: Uploading piece... Dec 27 18:18:48.978629147 (530h49m48.849712182s) [noise=true , address=145.79.227.45:4230 , piece_id=BHEUT4KRORKUD3CFOCIYSYJPISFV3EQMNQJ7Y3FB3KSOGIDUAIJQ, seg_pos={0 0}, upload=465546]: Uploading piece... Dec 27 18:18:48.978701996 (530h49m48.849785031s) [noise=true , address=145.79.227.17:4972 , piece_id=NHQ6L2GHJA7UGARK2BA7HQNSNAFGILDJOCPQBDVWTOGLFXWDSERA, seg_pos={0 0}, upload=465546]: Uploading piece... Dec 27 18:18:48.978951932 (530h49m48.850034967s) [noise=true , address=45.91.116.9:4261 , piece_id=6KKONXYFBLEKXTJMNQ452QDLMGOFTGOML2W77L7AOCW6IB5D3ELQ, seg_pos={0 0}, upload=465546]: Uploading piece... Dec 27 18:18:48.979365720 (530h49m48.850448765s) [noise=true , address=77.90.31.14:4664 , piece_id=J42YQFOCQEVVX32GWSIDMOGZFXZ7KIO5KH5XUJ36XI3KVXYSKPNA, seg_pos={0 0}, upload=465546]: Uploading piece... Dec 27 18:18:48.979489626 (530h49m48.850572681s) [noise=true , address=77.90.31.19:4844 , piece_id=57WUY7F7ERDXIXGF5LLR5V5SD3CFEMZHEA4TZY7N2PVHWORCO2GQ, seg_pos={0 0}, upload=465546]: Uploading piece... Dec 27 18:18:48.979717109 (530h49m48.850800154s) [noise=true , address=77.90.31.13:4627 , piece_id=7IXYPYPEI77UG5VYCM6MGHA5JSEG6PGUUSJMDZ7BKWGB2V24MBSA, seg_pos={0 0}, upload=465546]: Uploading piece... Dec 27 18:18:48.980035897 (530h49m48.851118932s) [noise=true , address=145.79.227.52:4447 , piece_id=KB2TWKDOQFH7JQ24QNMUHWZLIFMO2UEINGNIVSELXYFEY36F5SMA, seg_pos={0 0}, upload=465546]: Uploading piece... Dec 27 18:18:48.980186343 (530h49m48.851269388s) [noise=true , address=45.91.116.4:4939 , piece_id=P2MFR7PJ6GJOENSTD2XRPYMTAUTGNQWNLCTVXLTFNZO3RMP7KDDQ, seg_pos={0 0}, upload=465546]: Uploading piece... Dec 27 18:18:48.981159686 (530h49m48.852242731s) [noise=true , address=145.79.227.37:4566 , piece_id=JBT3U7FWWPQXN7TTXCKP7TVWO5C52RN3PADBCYAN2KHH7S7D7NTA, seg_pos={0 0}, upload=465546]: Uploading piece... Dec 27 18:18:50.593473445 (530h49m50.46455649s) [upload=465546]: Next returned nil segment Dec 27 18:18:50.593485127 (530h49m50.464568162s) [upload=465546]: Waiting for error group managing segments... Dec 27 18:18:50.978438319 (530h49m50.849521364s) [noise=true , address=145.79.227.3:4151 , piece_id=V5F6ZDZ4S4EUAJI23OZTCWOD5QH5OHOWUENXFHIDNEJXNU6JA7AQ, seg_pos={0 0}, upload=465546]: Done uploading piece. err: <nil> Dec 27 18:18:50.983972487 (530h49m50.855055532s) [noise=true , address=145.79.227.37:4566 , piece_id=JBT3U7FWWPQXN7TTXCKP7TVWO5C52RN3PADBCYAN2KHH7S7D7NTA, seg_pos={0 0}, upload=465546]: Done uploading piece. err: <nil> Dec 27 18:18:50.987682160 (530h49m50.858765206s) [noise=true , address=145.79.227.7:4295 , piece_id=YUTDEZNBRFGICWDWL3UHOYHRIZH7RZSY43AAARQ7H72FVHV2TWRA, seg_pos={0 0}, upload=465546]: Done uploading piece. err: <nil> Dec 27 18:18:50.988931730 (530h49m50.860014775s) [noise=true , address=145.79.227.25:4618 , piece_id=657IUHGFLLOLMJVNEWFYZ55FLFQI5QKFO7ENAH6GKYDPI7GQVXFQ, seg_pos={0 0}, upload=465546]: Done uploading piece. err: <nil> Dec 27 18:18:50.991081203 (530h49m50.862164238s) [noise=true , address=145.79.227.52:4447 , piece_id=KB2TWKDOQFH7JQ24QNMUHWZLIFMO2UEINGNIVSELXYFEY36F5SMA, seg_pos={0 0}, upload=465546]: Done uploading piece. err: <nil> Dec 27 18:18:50.999632427 (530h49m50.870715472s) [noise=true , address=145.79.227.45:4230 , piece_id=BHEUT4KRORKUD3CFOCIYSYJPISFV3EQMNQJ7Y3FB3KSOGIDUAIJQ, seg_pos={0 0}, upload=465546]: Done uploading piece. err: <nil> Dec 27 18:18:51.006550861 (530h49m50.877633896s) [noise=true , address=145.79.227.17:4972 , piece_id=NHQ6L2GHJA7UGARK2BA7HQNSNAFGILDJOCPQBDVWTOGLFXWDSERA, seg_pos={0 0}, upload=465546]: Done uploading piece. err: <nil> Dec 27 18:18:51.027665085 (530h49m50.89874813s) [noise=true , address=145.79.227.47:4265 , piece_id=T4WQNGBMENUE32AYQ2MXFAKR3LRYIOVRCJJE37V2XOKOJ4VVIJYA, seg_pos={0 0}, upload=465546]: Done uploading piece. err: <nil> Dec 27 18:18:51.146239203 (530h49m51.017322248s) [noise=true , address=77.90.31.14:4664 , piece_id=J42YQFOCQEVVX32GWSIDMOGZFXZ7KIO5KH5XUJ36XI3KVXYSKPNA, seg_pos={0 0}, upload=465546]: Done uploading piece. err: <nil> Dec 27 18:18:51.164235901 (530h49m51.035318936s) [noise=true , address=77.90.31.10:4519 , piece_id=EUF7QLG6CJGERWWATE2N7QGNETUZYWDTFOC6LQVXUFQLKZKZSH5Q, seg_pos={0 0}, upload=465546]: Done uploading piece. err: <nil> Dec 27 18:18:51.231133206 (530h49m51.102216241s) [noise=true , address=45.91.116.9:4261 , piece_id=6KKONXYFBLEKXTJMNQ452QDLMGOFTGOML2W77L7AOCW6IB5D3ELQ, seg_pos={0 0}, upload=465546]: Done uploading piece. err: <nil> Dec 27 18:18:51.255307578 (530h49m51.126390623s) [noise=true , address=77.90.31.13:4627 , piece_id=7IXYPYPEI77UG5VYCM6MGHA5JSEG6PGUUSJMDZ7BKWGB2V24MBSA, seg_pos={0 0}, upload=465546]: Done uploading piece. err: <nil> Dec 27 18:18:51.282041364 (530h49m51.153124409s) [noise=true , address=77.90.31.19:4844 , piece_id=57WUY7F7ERDXIXGF5LLR5V5SD3CFEMZHEA4TZY7N2PVHWORCO2GQ, seg_pos={0 0}, upload=465546]: Done uploading piece. err: <nil> Dec 27 18:18:51.282068656 (530h49m51.153151701s) [seg_pos={0 0}, upload=465546]: Segment reached optimal threshold of 13 pieces. Dec 27 18:18:51.282286801 (530h49m51.153369847s) [noise=true , address=77.90.31.12:4589 , piece_id=3MI7AOPYUTKLWMURIBUKOD7DNP4HYUPST63WDVZXLSEYPGFILXKA, seg_pos={0 0}, upload=465546]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv): protocol: write tcp 188.166.225.170:39916->77.90.31.12:4589: use of closed network connection; write tcp 188.166.225.170:39916->77.90.31.12:4589: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39916->77.90.31.12:4589: use of closed network connection Dec 27 18:18:51.282294697 (530h49m51.153377742s) [noise=true , address=77.90.31.12:4589 , piece_id=3MI7AOPYUTKLWMURIBUKOD7DNP4HYUPST63WDVZXLSEYPGFILXKA, seg_pos={0 0}, upload=465546]: Long tail cancellation due to optimal threshold Dec 27 18:18:51.282436467 (530h49m51.153519522s) [noise=true , address=45.91.116.4:4939 , piece_id=P2MFR7PJ6GJOENSTD2XRPYMTAUTGNQWNLCTVXLTFNZO3RMP7KDDQ, seg_pos={0 0}, upload=465546]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: write tcp 188.166.225.170:55750->45.91.116.4:4939: use of closed network connection; write tcp 188.166.225.170:55750->45.91.116.4:4939: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55750->45.91.116.4:4939: use of closed network connection Dec 27 18:18:51.282606351 (530h49m51.153689386s) [noise=true , address=45.91.116.4:4939 , piece_id=P2MFR7PJ6GJOENSTD2XRPYMTAUTGNQWNLCTVXLTFNZO3RMP7KDDQ, seg_pos={0 0}, upload=465546]: Long tail cancellation due to optimal threshold Dec 27 18:18:51.282667006 (530h49m51.153750051s) [noise=true , address=45.91.116.7:4010 , piece_id=EY2J4FJ4JPQ2I2EK324SJ2Q43TOYQYPLJWXOUJNCRTCXXOZKMATA, seg_pos={0 0}, upload=465546]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: expected piece hash; context canceled; EOF Dec 27 18:18:51.282676334 (530h49m51.153759379s) [noise=true , address=45.91.116.7:4010 , piece_id=EY2J4FJ4JPQ2I2EK324SJ2Q43TOYQYPLJWXOUJNCRTCXXOZKMATA, seg_pos={0 0}, upload=465546]: Long tail cancellation due to optimal threshold Dec 27 18:18:51.282764662 (530h49m51.153847707s) [seg_pos={0 0}, upload=465546]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 18:18:51.282917964 (530h49m51.154001009s) [upload=465546]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 18:18:51.282924527 (530h49m51.154007572s) [upload=465546]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 18:18:51.598820670 (530h49m51.469903705s) [upload=465546]: Done uploading segments... Dec 27 18:18:51.598840398 (530h49m51.469923433s) [upload=465546]: Upload finished. err: <nil> Duration: 30.002606992s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [............X...............................................] duration: 3.613767162s trace: https://jaeger.dev.storj.io/trace/3bddfef0dfa96557 FAIL (2025-12-27 18:18:55 UTC) orbiter warning: context deadline exceeded Duration: 3.61381345s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 24 out of 60 is failed (threshold=3) [X.X..XXXX.X....XXX..XX....X...X.X........X.......X..X.XXXXXX] duration: 30.002597784s trace: https://jaeger.dev.storj.io/trace/5b4659c7e613cd57 FAIL (2025-12-27 18:18:47 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG, Piece ID: RIEDDTGKTNYGJZ5PKXS65FW4ZPNBRE37KSJY7MBMKXNEV7OJTFOQ) context deadline exceeded; piecestore: (Node ID: 1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, Piece ID: GYOCSOGUUUATXIZOSTBUFX33KOT767SBDTMOWM6MBJ2JVTH3PDLA) context deadline exceeded ======LOGS====== nts... Dec 27 18:18:18.491353699 (530h49m18.362436734s) [upload=465540]: Got next segment. Inline: false Dec 27 18:18:18.491402692 (530h49m18.362485757s) [upload=465540]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 18:18:18.491408723 (530h49m18.362491768s) [upload=465540]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 18:18:18.736247932 (530h49m18.607330977s) [seg_pos={0 0}, upload=465540]: Begin upload segment... Dec 27 18:18:18.736335028 (530h49m18.607418073s) [seg_pos={0 0}, upload=465540]: Done begin upload segment. Dec 27 18:18:18.736377780 (530h49m18.607460815s) [noise=true , address=145.79.227.20:4438 , piece_id=AGINRHCVMZKPFCODYFQLY2JYCJC6HIUJCBYS7IZH52YX4QNSD77A, seg_pos={0 0}, upload=465540]: Uploading piece... Dec 27 18:18:18.736405072 (530h49m18.607488117s) [noise=true , address=45.91.116.4:4939 , piece_id=WLWISOR624ZQIC5B6U3RIVBPELBMLIN6NFSDCROHMXEMAFYNBKEA, seg_pos={0 0}, upload=465540]: Uploading piece... Dec 27 18:18:18.737052955 (530h49m18.60813599s) [noise=true , address=45.91.116.10:4081 , piece_id=RIEDDTGKTNYGJZ5PKXS65FW4ZPNBRE37KSJY7MBMKXNEV7OJTFOQ, seg_pos={0 0}, upload=465540]: Uploading piece... Dec 27 18:18:18.737142406 (530h49m18.608225451s) [noise=true , address=45.91.116.8:4045 , piece_id=GYOCSOGUUUATXIZOSTBUFX33KOT767SBDTMOWM6MBJ2JVTH3PDLA, seg_pos={0 0}, upload=465540]: Uploading piece... Dec 27 18:18:18.737616077 (530h49m18.608699122s) [noise=true , address=145.79.227.14:4839 , piece_id=LZR55FEJOK4QGQQLJCPMC2K243LS7AIP2ZPI6N2LOVJ4N6J5P3HA, seg_pos={0 0}, upload=465540]: Uploading piece... Dec 27 18:18:18.738073247 (530h49m18.609156282s) [noise=true , address=145.79.227.28:4727 , piece_id=RSN7RCVQGONZN3OZL2IDRVQ4EFH3TOZZPY2OPVITFZKJKO4GVKBQ, seg_pos={0 0}, upload=465540]: Uploading piece... Dec 27 18:18:18.738322883 (530h49m18.609405929s) [noise=true , address=145.79.227.50:4375 , piece_id=DKJHVGZEFSGFYBYLNXTZDS5BCKDKUPI46W2IB2BXMAJ46DDT533Q, seg_pos={0 0}, upload=465540]: Uploading piece... Dec 27 18:18:18.738634226 (530h49m18.609717261s) [noise=true , address=77.90.31.19:4842 , piece_id=K44L4UJ3LNSHESWNV7LZ65TIJFB5PYFS3B76INRAW2HIQ5QHANHA, seg_pos={0 0}, upload=465540]: Uploading piece... Dec 27 18:18:18.738808568 (530h49m18.609891613s) [noise=true , address=145.79.227.8:4329 , piece_id=UBINTOUXOZCCJIR4VIQ7DU6536QJSUIJR2LDR2BG254MEHBEUPKQ, seg_pos={0 0}, upload=465540]: Uploading piece... Dec 27 18:18:18.739089363 (530h49m18.610172408s) [noise=true , address=145.79.227.11:4732 , piece_id=GIVU5AU4XFBTKGWUWPNYT4WHQEFHGFT55WSUDMSK5VIX7QTNKTTQ, seg_pos={0 0}, upload=465540]: Uploading piece... Dec 27 18:18:18.739130150 (530h49m18.610213185s) [noise=true , address=45.91.116.11:4117 , piece_id=FWKXGCKHDT5CHU2RXT5CBS7NIBHX5NZIB4L2IUJVU5WDXSRQCIPA, seg_pos={0 0}, upload=465540]: Uploading piece... Dec 27 18:18:18.739508321 (530h49m18.610591356s) [noise=true , address=145.79.227.17:4971 , piece_id=6QBLQDP5W7G7GP754PBTV3NXUSK5DF7F76UJWQVCZJXZZ6EVE6OQ, seg_pos={0 0}, upload=465540]: Uploading piece... Dec 27 18:18:18.739544770 (530h49m18.610627805s) [noise=true , address=145.79.227.2:4114 , piece_id=BPGZZMY64EUWH4IIB735XKRBNMQ3NN2WV6LQ2QKOH4HUSOPXGI5A, seg_pos={0 0}, upload=465540]: Uploading piece... Dec 27 18:18:18.739796189 (530h49m18.610879224s) [noise=true , address=45.91.116.6:4973 , piece_id=Q25VZBAUV7ZLHNKLHUBBAUCPELSN43NQPIRARPUK3GKCDYP5EUJA, seg_pos={0 0}, upload=465540]: Uploading piece... Dec 27 18:18:18.740012420 (530h49m18.611095455s) [noise=true , address=145.79.227.39:4014 , piece_id=6YBR6DPCGMGITVRGSK4P4XGXWSZINKYK5KOL275GIBPHA5WBIF7Q, seg_pos={0 0}, upload=465540]: Uploading piece... Dec 27 18:18:18.740106600 (530h49m18.611189645s) [noise=true , address=145.79.227.52:4445 , piece_id=OBMW7W5OUIK26KHFQ35P3ZZBRODCTPIQ3KI5OI3LSJEQHQFMTASQ, seg_pos={0 0}, upload=465540]: Uploading piece... Dec 27 18:18:20.444066038 (530h49m20.315149083s) [upload=465540]: Next returned nil segment Dec 27 18:18:20.444079033 (530h49m20.315162068s) [upload=465540]: Waiting for error group managing segments... Dec 27 18:18:20.727202522 (530h49m20.598285567s) [noise=true , address=145.79.227.14:4839 , piece_id=LZR55FEJOK4QGQQLJCPMC2K243LS7AIP2ZPI6N2LOVJ4N6J5P3HA, seg_pos={0 0}, upload=465540]: Done uploading piece. err: <nil> Dec 27 18:18:20.727469550 (530h49m20.598552585s) [noise=true , address=145.79.227.52:4445 , piece_id=OBMW7W5OUIK26KHFQ35P3ZZBRODCTPIQ3KI5OI3LSJEQHQFMTASQ, seg_pos={0 0}, upload=465540]: Done uploading piece. err: <nil> Dec 27 18:18:20.735557541 (530h49m20.606640586s) [noise=true , address=145.79.227.20:4438 , piece_id=AGINRHCVMZKPFCODYFQLY2JYCJC6HIUJCBYS7IZH52YX4QNSD77A, seg_pos={0 0}, upload=465540]: Done uploading piece. err: <nil> Dec 27 18:18:20.736738400 (530h49m20.607821445s) [noise=true , address=145.79.227.2:4114 , piece_id=BPGZZMY64EUWH4IIB735XKRBNMQ3NN2WV6LQ2QKOH4HUSOPXGI5A, seg_pos={0 0}, upload=465540]: Done uploading piece. err: <nil> Dec 27 18:18:20.742784853 (530h49m20.613867888s) [noise=true , address=145.79.227.28:4727 , piece_id=RSN7RCVQGONZN3OZL2IDRVQ4EFH3TOZZPY2OPVITFZKJKO4GVKBQ, seg_pos={0 0}, upload=465540]: Done uploading piece. err: <nil> Dec 27 18:18:20.743686721 (530h49m20.614769756s) [noise=true , address=145.79.227.39:4014 , piece_id=6YBR6DPCGMGITVRGSK4P4XGXWSZINKYK5KOL275GIBPHA5WBIF7Q, seg_pos={0 0}, upload=465540]: Done uploading piece. err: <nil> Dec 27 18:18:20.749983461 (530h49m20.621066506s) [noise=true , address=145.79.227.17:4971 , piece_id=6QBLQDP5W7G7GP754PBTV3NXUSK5DF7F76UJWQVCZJXZZ6EVE6OQ, seg_pos={0 0}, upload=465540]: Done uploading piece. err: <nil> Dec 27 18:18:20.766078810 (530h49m20.637161855s) [noise=true , address=145.79.227.50:4375 , piece_id=DKJHVGZEFSGFYBYLNXTZDS5BCKDKUPI46W2IB2BXMAJ46DDT533Q, seg_pos={0 0}, upload=465540]: Done uploading piece. err: <nil> Dec 27 18:18:20.778041508 (530h49m20.649124543s) [noise=true , address=145.79.227.8:4329 , piece_id=UBINTOUXOZCCJIR4VIQ7DU6536QJSUIJR2LDR2BG254MEHBEUPKQ, seg_pos={0 0}, upload=465540]: Done uploading piece. err: <nil> Dec 27 18:18:20.791689316 (530h49m20.662772361s) [noise=true , address=145.79.227.11:4732 , piece_id=GIVU5AU4XFBTKGWUWPNYT4WHQEFHGFT55WSUDMSK5VIX7QTNKTTQ, seg_pos={0 0}, upload=465540]: Done uploading piece. err: <nil> Dec 27 18:18:21.070354366 (530h49m20.941437411s) [noise=true , address=45.91.116.8:4045 , piece_id=GYOCSOGUUUATXIZOSTBUFX33KOT767SBDTMOWM6MBJ2JVTH3PDLA, seg_pos={0 0}, upload=465540]: Done uploading piece. err: <nil> Dec 27 18:18:21.083512792 (530h49m20.954595837s) [noise=true , address=45.91.116.10:4081 , piece_id=RIEDDTGKTNYGJZ5PKXS65FW4ZPNBRE37KSJY7MBMKXNEV7OJTFOQ, seg_pos={0 0}, upload=465540]: Done uploading piece. err: <nil> Dec 27 18:18:21.183765732 (530h49m21.054848777s) [noise=true , address=45.91.116.6:4973 , piece_id=Q25VZBAUV7ZLHNKLHUBBAUCPELSN43NQPIRARPUK3GKCDYP5EUJA, seg_pos={0 0}, upload=465540]: Done uploading piece. err: <nil> Dec 27 18:18:21.183785700 (530h49m21.054868745s) [seg_pos={0 0}, upload=465540]: Segment reached optimal threshold of 13 pieces. Dec 27 18:18:21.184085170 (530h49m21.055168225s) [noise=true , address=45.91.116.11:4117 , piece_id=FWKXGCKHDT5CHU2RXT5CBS7NIBHX5NZIB4L2IUJVU5WDXSRQCIPA, seg_pos={0 0}, upload=465540]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6): protocol: write tcp 188.166.225.170:54560->45.91.116.11:4117: use of closed network connection; write tcp 188.166.225.170:54560->45.91.116.11:4117: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54560->45.91.116.11:4117: use of closed network connection Dec 27 18:18:21.184094017 (530h49m21.055177062s) [noise=true , address=45.91.116.11:4117 , piece_id=FWKXGCKHDT5CHU2RXT5CBS7NIBHX5NZIB4L2IUJVU5WDXSRQCIPA, seg_pos={0 0}, upload=465540]: Long tail cancellation due to optimal threshold Dec 27 18:18:21.184186964 (530h49m21.055270009s) [noise=true , address=77.90.31.19:4842 , piece_id=K44L4UJ3LNSHESWNV7LZ65TIJFB5PYFS3B76INRAW2HIQ5QHANHA, seg_pos={0 0}, upload=465540]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: expected piece hash; context canceled; EOF Dec 27 18:18:21.184200730 (530h49m21.055283765s) [noise=true , address=77.90.31.19:4842 , piece_id=K44L4UJ3LNSHESWNV7LZ65TIJFB5PYFS3B76INRAW2HIQ5QHANHA, seg_pos={0 0}, upload=465540]: Long tail cancellation due to optimal threshold Dec 27 18:18:21.184432932 (530h49m21.055515967s) [noise=true , address=45.91.116.4:4939 , piece_id=WLWISOR624ZQIC5B6U3RIVBPELBMLIN6NFSDCROHMXEMAFYNBKEA, seg_pos={0 0}, upload=465540]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: write tcp 188.166.225.170:52058->45.91.116.4:4939: use of closed network connection; write tcp 188.166.225.170:52058->45.91.116.4:4939: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52058->45.91.116.4:4939: use of closed network connection Dec 27 18:18:21.184443983 (530h49m21.055527018s) [noise=true , address=45.91.116.4:4939 , piece_id=WLWISOR624ZQIC5B6U3RIVBPELBMLIN6NFSDCROHMXEMAFYNBKEA, seg_pos={0 0}, upload=465540]: Long tail cancellation due to optimal threshold Dec 27 18:18:21.184468320 (530h49m21.055551365s) [seg_pos={0 0}, upload=465540]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 18:18:21.184609388 (530h49m21.055692433s) [upload=465540]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 18:18:21.184616882 (530h49m21.055699927s) [upload=465540]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 18:18:21.483976558 (530h49m21.355059593s) [upload=465540]: Done uploading segments... Dec 27 18:18:21.484001997 (530h49m21.355085042s) [upload=465540]: Upload finished. err: <nil> Duration: 30.002620698s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.............X..............................................] duration: 3.529841912s trace: https://jaeger.dev.storj.io/trace/3bddfef0dfa96557 FAIL (2025-12-27 18:18:20 UTC) orbiter warning: context deadline exceeded Duration: 3.52987719s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..............X.............................................] duration: 3.596999995s trace: https://jaeger.dev.storj.io/trace/3bddfef0dfa96557 FAIL (2025-12-27 18:18:05 UTC) orbiter warning: context deadline exceeded Duration: 3.597029121s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 23 out of 60 is failed (threshold=3) [.X.X..XXXX.X....XXX..XX....X...X.X........X.......X..X.XXXXX] duration: 30.003293218s trace: https://jaeger.dev.storj.io/trace/1052af1d0e62e357 FAIL (2025-12-27 18:17:36 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 465521]: Starting upload Dec 27 18:17:07.015090265 (530h48m6.88617331s) [upload=465521]: Done starting upload Dec 27 18:17:07.015216264 (530h48m6.886299299s) [upload=465521]: Uploading segments... Dec 27 18:17:07.015322516 (530h48m6.886405551s) [upload=465521]: Got next segment. Inline: false Dec 27 18:17:07.015356591 (530h48m6.886439626s) [upload=465521]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 18:17:07.015363054 (530h48m6.886446099s) [upload=465521]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 18:17:07.262170781 (530h48m7.133253816s) [seg_pos={0 0}, upload=465521]: Begin upload segment... Dec 27 18:17:07.262216527 (530h48m7.133299573s) [seg_pos={0 0}, upload=465521]: Done begin upload segment. Dec 27 18:17:07.262233920 (530h48m7.133316965s) [noise=true , address=45.91.116.8:4048 , piece_id=EYXHVA2Z2RUREE6SATUWBW3NNN336GE743KRBDAUCEF4MTJBKWJA, seg_pos={0 0}, upload=465521]: Uploading piece... Dec 27 18:17:07.262245673 (530h48m7.133328708s) [noise=true , address=145.79.227.14:4838 , piece_id=TZ7BIFSPIRG3GOJ73VNYY44CSGKBTDKM72NDSNTO7UI7ZFEFGQ7Q, seg_pos={0 0}, upload=465521]: Uploading piece... Dec 27 18:17:07.262890721 (530h48m7.133973756s) [noise=true , address=45.91.116.7:4011 , piece_id=RTLANRN54FO3ONQXP3NKX5GFZ3KDXLSMDOHCH7EGT6Y67ETRBWHA, seg_pos={0 0}, upload=465521]: Uploading piece... Dec 27 18:17:07.263400332 (530h48m7.134483377s) [noise=true , address=145.79.227.50:4375 , piece_id=ABVM3DB2BPM5UR5CIVQ7LWR7777T2BP7IZBSN6ZEW32M6U6FFQZA, seg_pos={0 0}, upload=465521]: Uploading piece... Dec 27 18:17:07.263663253 (530h48m7.134746298s) [noise=true , address=77.90.31.16:4735 , piece_id=QPO4RPAYEROUVNVU234EHYISBEGIHIBD65FBCZ5XRVS6QCI2ZYWQ, seg_pos={0 0}, upload=465521]: Uploading piece... Dec 27 18:17:07.265395942 (530h48m7.136479007s) [noise=true , address=145.79.227.38:4977 , piece_id=6HXJI4YBX5KPUMAQSTMQAMJOPIL4P2ZQSZ5XLOCOLCJNXXHR25WA, seg_pos={0 0}, upload=465521]: Uploading piece... Dec 27 18:17:07.265878180 (530h48m7.136961215s) [noise=true , address=45.91.116.13:4154 , piece_id=5DZ7SAYKG5GQAXWL262WLFJXSR2XE2MUKR25O34H2FDHTIVI7YOQ, seg_pos={0 0}, upload=465521]: Uploading piece... Dec 27 18:17:07.267988629 (530h48m7.139071664s) [noise=true , address=45.91.116.5:4300 , piece_id=RYLJOW5TUWYXSORJGH3V22XMURNRTFDSBVN3JUD4OG4E5TIPI5TA, seg_pos={0 0}, upload=465521]: Uploading piece... Dec 27 18:17:07.269188344 (530h48m7.140271389s) [noise=true , address=145.79.227.28:4726 , piece_id=VSWGY6OLJCDWJSBCDQEHZS2YG32M5VTWSQ56T4VNODOMOWBT2YMQ, seg_pos={0 0}, upload=465521]: Uploading piece... Dec 27 18:17:07.270284761 (530h48m7.141367806s) [noise=true , address=145.79.227.11:4730 , piece_id=62GJCVVSEZI5GOOXWX6RZ7WJANBKHGW3ZCK5EWSQVYBXCCBJ5N2Q, seg_pos={0 0}, upload=465521]: Uploading piece... Dec 27 18:17:07.271407949 (530h48m7.142490995s) [noise=true , address=145.79.227.37:4566 , piece_id=SC4MJHWKP4AOBQG7B7NXLLJHA2ACRGDSAMC2WMXSOKTKP5THL2IA, seg_pos={0 0}, upload=465521]: Uploading piece... Dec 27 18:17:07.271965752 (530h48m7.143048817s) [noise=true , address=45.91.116.9:4263 , piece_id=R2RKSCBXN3DTPGFJSMDMUTXZFBGRTSED55WIB3CS67IREKYCUFKQ, seg_pos={0 0}, upload=465521]: Uploading piece... Dec 27 18:17:07.273364075 (530h48m7.14444712s) [noise=true , address=77.90.31.17:4773 , piece_id=IRWYVPKRBPCTQONR755HIFK6S6WCNICZAKYLL2XTVOHH45KFTZYA, seg_pos={0 0}, upload=465521]: Uploading piece... Dec 27 18:17:07.274248439 (530h48m7.145331484s) [noise=true , address=145.79.227.15:4874 , piece_id=52JPEKZDSO4PRW7QAWYYWCJKDWIZLIWA46TMDW6DMKG467G525SA, seg_pos={0 0}, upload=465521]: Uploading piece... Dec 27 18:17:07.275163982 (530h48m7.146247027s) [noise=true , address=145.79.227.42:4123 , piece_id=5Z7Y4LV4JZSMO2UND37YVRKR3SHVQBFS4K5XNOUGTTV6YPZBBQUQ, seg_pos={0 0}, upload=465521]: Uploading piece... Dec 27 18:17:07.276835434 (530h48m7.147918469s) [noise=true , address=145.79.227.32:4834 , piece_id=ZPOZLIFZ544XDTPZJ7N6ODBDPWDLX75E3N2L6WROGF5FBP3VDZOA, seg_pos={0 0}, upload=465521]: Uploading piece... Dec 27 18:17:08.877609861 (530h48m8.748692896s) [upload=465521]: Next returned nil segment Dec 27 18:17:08.877624570 (530h48m8.748707615s) [upload=465521]: Waiting for error group managing segments... Dec 27 18:17:09.262385694 (530h48m9.133468739s) [noise=true , address=145.79.227.14:4838 , piece_id=TZ7BIFSPIRG3GOJ73VNYY44CSGKBTDKM72NDSNTO7UI7ZFEFGQ7Q, seg_pos={0 0}, upload=465521]: Done uploading piece. err: <nil> Dec 27 18:17:09.266501982 (530h48m9.137585027s) [noise=true , address=145.79.227.42:4123 , piece_id=5Z7Y4LV4JZSMO2UND37YVRKR3SHVQBFS4K5XNOUGTTV6YPZBBQUQ, seg_pos={0 0}, upload=465521]: Done uploading piece. err: <nil> Dec 27 18:17:09.267329899 (530h48m9.138412934s) [noise=true , address=145.79.227.50:4375 , piece_id=ABVM3DB2BPM5UR5CIVQ7LWR7777T2BP7IZBSN6ZEW32M6U6FFQZA, seg_pos={0 0}, upload=465521]: Done uploading piece. err: <nil> Dec 27 18:17:09.267913531 (530h48m9.138996576s) [noise=true , address=145.79.227.15:4874 , piece_id=52JPEKZDSO4PRW7QAWYYWCJKDWIZLIWA46TMDW6DMKG467G525SA, seg_pos={0 0}, upload=465521]: Done uploading piece. err: <nil> Dec 27 18:17:09.269919189 (530h48m9.141002234s) [noise=true , address=145.79.227.32:4834 , piece_id=ZPOZLIFZ544XDTPZJ7N6ODBDPWDLX75E3N2L6WROGF5FBP3VDZOA, seg_pos={0 0}, upload=465521]: Done uploading piece. err: <nil> Dec 27 18:17:09.271956930 (530h48m9.143039975s) [noise=true , address=145.79.227.11:4730 , piece_id=62GJCVVSEZI5GOOXWX6RZ7WJANBKHGW3ZCK5EWSQVYBXCCBJ5N2Q, seg_pos={0 0}, upload=465521]: Done uploading piece. err: <nil> Dec 27 18:17:09.275666283 (530h48m9.146749329s) [noise=true , address=145.79.227.38:4977 , piece_id=6HXJI4YBX5KPUMAQSTMQAMJOPIL4P2ZQSZ5XLOCOLCJNXXHR25WA, seg_pos={0 0}, upload=465521]: Done uploading piece. err: <nil> Dec 27 18:17:09.293240187 (530h48m9.164323222s) [noise=true , address=145.79.227.37:4566 , piece_id=SC4MJHWKP4AOBQG7B7NXLLJHA2ACRGDSAMC2WMXSOKTKP5THL2IA, seg_pos={0 0}, upload=465521]: Done uploading piece. err: <nil> Dec 27 18:17:09.295782919 (530h48m9.166865954s) [noise=true , address=145.79.227.28:4726 , piece_id=VSWGY6OLJCDWJSBCDQEHZS2YG32M5VTWSQ56T4VNODOMOWBT2YMQ, seg_pos={0 0}, upload=465521]: Done uploading piece. err: <nil> Dec 27 18:17:09.495405466 (530h48m9.366488501s) [noise=true , address=45.91.116.9:4263 , piece_id=R2RKSCBXN3DTPGFJSMDMUTXZFBGRTSED55WIB3CS67IREKYCUFKQ, seg_pos={0 0}, upload=465521]: Done uploading piece. err: <nil> Dec 27 18:17:09.559514432 (530h48m9.430597467s) [noise=true , address=77.90.31.16:4735 , piece_id=QPO4RPAYEROUVNVU234EHYISBEGIHIBD65FBCZ5XRVS6QCI2ZYWQ, seg_pos={0 0}, upload=465521]: Done uploading piece. err: <nil> Dec 27 18:17:09.574652658 (530h48m9.445735693s) [noise=true , address=45.91.116.13:4154 , piece_id=5DZ7SAYKG5GQAXWL262WLFJXSR2XE2MUKR25O34H2FDHTIVI7YOQ, seg_pos={0 0}, upload=465521]: Done uploading piece. err: <nil> Dec 27 18:17:09.609507127 (530h48m9.480590172s) [noise=true , address=45.91.116.5:4300 , piece_id=RYLJOW5TUWYXSORJGH3V22XMURNRTFDSBVN3JUD4OG4E5TIPI5TA, seg_pos={0 0}, upload=465521]: Done uploading piece. err: <nil> Dec 27 18:17:09.609528308 (530h48m9.480611343s) [seg_pos={0 0}, upload=465521]: Segment reached optimal threshold of 13 pieces. Dec 27 18:17:09.609738017 (530h48m9.480821052s) [noise=true , address=77.90.31.17:4773 , piece_id=IRWYVPKRBPCTQONR755HIFK6S6WCNICZAKYLL2XTVOHH45KFTZYA, seg_pos={0 0}, upload=465521]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ): protocol: write tcp 188.166.225.170:48428->77.90.31.17:4773: use of closed network connection; write tcp 188.166.225.170:48428->77.90.31.17:4773: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48428->77.90.31.17:4773: use of closed network connection Dec 27 18:17:09.609745571 (530h48m9.480828616s) [noise=true , address=77.90.31.17:4773 , piece_id=IRWYVPKRBPCTQONR755HIFK6S6WCNICZAKYLL2XTVOHH45KFTZYA, seg_pos={0 0}, upload=465521]: Long tail cancellation due to optimal threshold Dec 27 18:17:09.610085008 (530h48m9.481168053s) [noise=true , address=45.91.116.7:4011 , piece_id=RTLANRN54FO3ONQXP3NKX5GFZ3KDXLSMDOHCH7EGT6Y67ETRBWHA, seg_pos={0 0}, upload=465521]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf): protocol: read tcp 188.166.225.170:37048->45.91.116.7:4011: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 27 18:17:09.610103041 (530h48m9.481186077s) [noise=true , address=45.91.116.7:4011 , piece_id=RTLANRN54FO3ONQXP3NKX5GFZ3KDXLSMDOHCH7EGT6Y67ETRBWHA, seg_pos={0 0}, upload=465521]: Long tail cancellation due to optimal threshold Dec 27 18:17:09.610167835 (530h48m9.48125087s) [noise=true , address=45.91.116.8:4048 , piece_id=EYXHVA2Z2RUREE6SATUWBW3NNN336GE743KRBDAUCEF4MTJBKWJA, seg_pos={0 0}, upload=465521]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: expected piece hash; context canceled; EOF Dec 27 18:17:09.610176632 (530h48m9.481259678s) [noise=true , address=45.91.116.8:4048 , piece_id=EYXHVA2Z2RUREE6SATUWBW3NNN336GE743KRBDAUCEF4MTJBKWJA, seg_pos={0 0}, upload=465521]: Long tail cancellation due to optimal threshold Dec 27 18:17:09.610204645 (530h48m9.48128769s) [seg_pos={0 0}, upload=465521]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 18:17:09.610278435 (530h48m9.48136148s) [upload=465521]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 18:17:09.610282704 (530h48m9.481365749s) [upload=465521]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 18:17:09.896924453 (530h48m9.768007509s) [upload=465521]: Done uploading segments... Dec 27 18:17:09.896945012 (530h48m9.768028047s) [upload=465521]: Upload finished. err: <nil> Duration: 30.003311954s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...............X............................................] duration: 3.573784618s trace: https://jaeger.dev.storj.io/trace/3bddfef0dfa96557 FAIL (2025-12-27 18:17:07 UTC) orbiter warning: context deadline exceeded Duration: 3.573842018s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 23 out of 60 is failed (threshold=3) [X.X.X..XXXX.X....XXX..XX....X...X.X........X.......X..X.XXXX] duration: 30.00423068s trace: https://jaeger.dev.storj.io/trace/757969d2cd84dd00 FAIL (2025-12-27 18:16:34 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ, Piece ID: LJCIHM3FZNFVU4O4ZV4MWC6SKNGVDCYBXAVU7XD3C64ZLJGN3N3Q) context deadline exceeded; piecestore: (Node ID: 12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, Piece ID: DE3PLUNUBWJD6RE5AARM5SDPWP3U32MFFBEDPR2USHXAUCYPXI5A) context deadline exceeded ======LOGS====== Dec 27 18:16:05.344283394 (530h47m5.215366439s) [upload=465506]: Starting upload Dec 27 18:16:05.344338429 (530h47m5.215421464s) [upload=465506]: Done starting upload Dec 27 18:16:05.344483014 (530h47m5.215566059s) [upload=465506]: Uploading segments... Dec 27 18:16:05.344619965 (530h47m5.215703s) [upload=465506]: Got next segment. Inline: false Dec 27 18:16:05.344670161 (530h47m5.215753206s) [upload=465506]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 18:16:05.344675441 (530h47m5.215758496s) [upload=465506]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 18:16:05.590055691 (530h47m5.461138736s) [seg_pos={0 0}, upload=465506]: Begin upload segment... Dec 27 18:16:05.590113772 (530h47m5.461196807s) [seg_pos={0 0}, upload=465506]: Done begin upload segment. Dec 27 18:16:05.590151454 (530h47m5.461234489s) [noise=true , address=77.90.31.17:4773 , piece_id=LJCIHM3FZNFVU4O4ZV4MWC6SKNGVDCYBXAVU7XD3C64ZLJGN3N3Q, seg_pos={0 0}, upload=465506]: Uploading piece... Dec 27 18:16:05.590160130 (530h47m5.461243175s) [noise=true , address=145.79.227.48:4303 , piece_id=P5D5X5ORIZBYHDEIFC6TLCRVAGSPFNUKI5RTOUWULEYH6FTU4Y3Q, seg_pos={0 0}, upload=465506]: Uploading piece... Dec 27 18:16:05.590260161 (530h47m5.461343206s) [noise=true , address=145.79.227.15:4874 , piece_id=4FASBXBJPFNFFAETNGIKINVEHL76275PN2UXLWZS3WG3DCUO6RPA, seg_pos={0 0}, upload=465506]: Uploading piece... Dec 27 18:16:05.590798556 (530h47m5.461881601s) [noise=true , address=145.79.227.12:4765 , piece_id=7NIDT3MEPQBEPLIWYTX76YSDZ4RBCTNK7JNWRLHXIE3RJDHJ44PQ, seg_pos={0 0}, upload=465506]: Uploading piece... Dec 27 18:16:05.594884366 (530h47m5.465967401s) [noise=true , address=145.79.227.16:4911 , piece_id=EMTR4GXW55JY4E2RZ67AAF3IVAZZ2AR5T43BWCNCQ77XL5LRTW2A, seg_pos={0 0}, upload=465506]: Uploading piece... Dec 27 18:16:05.595319084 (530h47m5.466402129s) [noise=true , address=45.91.116.11:4120 , piece_id=Z5LI3ELXXRSIFZ33TOYWNXK6KFI6LB33SWZVBDIEQZA7WJKZQMAQ, seg_pos={0 0}, upload=465506]: Uploading piece... Dec 27 18:16:05.595701281 (530h47m5.466784326s) [noise=true , address=45.91.116.13:4153 , piece_id=RLOQLYRD5PCJ63NJZQ4OOZWMNJUPJTLY6GKQG4IMEKM75YGIA7HA, seg_pos={0 0}, upload=465506]: Uploading piece... Dec 27 18:16:05.596156879 (530h47m5.467239924s) [noise=true , address=145.79.227.5:4223 , piece_id=H7W3YYWUVSTHAJUKWRK74ADPKMKE4AFU6YUNXHHUIY5NZVILXCZA, seg_pos={0 0}, upload=465506]: Uploading piece... Dec 27 18:16:05.596551229 (530h47m5.467634275s) [noise=true , address=145.79.227.46:4602 , piece_id=EADSJHGMTKAO23CZPINBVQB7EAHQHJUKGR4OTNQPPTG2FGCVKK6Q, seg_pos={0 0}, upload=465506]: Uploading piece... Dec 27 18:16:05.596947114 (530h47m5.468030159s) [noise=true , address=145.79.227.4:4185 , piece_id=2YKY3G2AUAQDRNWVISS2HCAXK4QEL33MXPPRR6FN3NJZV6ZBX2LQ, seg_pos={0 0}, upload=465506]: Uploading piece... Dec 27 18:16:05.597367414 (530h47m5.468450459s) [noise=true , address=145.79.227.52:4445 , piece_id=YW254NXZDJKGZK32HZMZRA4RDB5QRWMEG74HDNM7OWISRV7ASDWQ, seg_pos={0 0}, upload=465506]: Uploading piece... Dec 27 18:16:05.598046316 (530h47m5.469129351s) [noise=true , address=77.90.31.10:4517 , piece_id=5UJOKHD6ZGY4XQWYAKTFHRBQZ6LHUWHFGQJZGLPHLGUQFTGLUTLA, seg_pos={0 0}, upload=465506]: Uploading piece... Dec 27 18:16:05.600463590 (530h47m5.471546625s) [noise=true , address=145.79.227.33:4869 , piece_id=KZRFP5EDEEFZYN5DWERCEGHIHLAXZDVNM6E4EF77ZZ6LIQTOWYEQ, seg_pos={0 0}, upload=465506]: Uploading piece... Dec 27 18:16:05.605348341 (530h47m5.476431376s) [noise=true , address=145.79.227.42:4123 , piece_id=6EQ5GKA3LAAHTW6KBS665BCVAT5DPU7LE7PEP2DEGO77AYOC5WHQ, seg_pos={0 0}, upload=465506]: Uploading piece... Dec 27 18:16:05.606974087 (530h47m5.478057172s) [noise=true , address=45.91.116.5:4299 , piece_id=DE3PLUNUBWJD6RE5AARM5SDPWP3U32MFFBEDPR2USHXAUCYPXI5A, seg_pos={0 0}, upload=465506]: Uploading piece... Dec 27 18:16:05.608368873 (530h47m5.479451908s) [noise=true , address=145.79.227.25:4619 , piece_id=KOMO72DG6NKMRJJCWK77R3EVK2MVP5CAGEQ4QV54DVTAVIKSDXYQ, seg_pos={0 0}, upload=465506]: Uploading piece... Dec 27 18:16:07.301127251 (530h47m7.172210297s) [upload=465506]: Next returned nil segment Dec 27 18:16:07.301150276 (530h47m7.172233321s) [upload=465506]: Waiting for error group managing segments... Dec 27 18:16:07.587049310 (530h47m7.458132355s) [noise=true , address=145.79.227.15:4874 , piece_id=4FASBXBJPFNFFAETNGIKINVEHL76275PN2UXLWZS3WG3DCUO6RPA, seg_pos={0 0}, upload=465506]: Done uploading piece. err: <nil> Dec 27 18:16:07.595408908 (530h47m7.466491953s) [noise=true , address=145.79.227.52:4445 , piece_id=YW254NXZDJKGZK32HZMZRA4RDB5QRWMEG74HDNM7OWISRV7ASDWQ, seg_pos={0 0}, upload=465506]: Done uploading piece. err: <nil> Dec 27 18:16:07.596246653 (530h47m7.467329698s) [noise=true , address=145.79.227.46:4602 , piece_id=EADSJHGMTKAO23CZPINBVQB7EAHQHJUKGR4OTNQPPTG2FGCVKK6Q, seg_pos={0 0}, upload=465506]: Done uploading piece. err: <nil> Dec 27 18:16:07.598299001 (530h47m7.469382036s) [noise=true , address=145.79.227.12:4765 , piece_id=7NIDT3MEPQBEPLIWYTX76YSDZ4RBCTNK7JNWRLHXIE3RJDHJ44PQ, seg_pos={0 0}, upload=465506]: Done uploading piece. err: <nil> Dec 27 18:16:07.607931092 (530h47m7.479014138s) [noise=true , address=145.79.227.48:4303 , piece_id=P5D5X5ORIZBYHDEIFC6TLCRVAGSPFNUKI5RTOUWULEYH6FTU4Y3Q, seg_pos={0 0}, upload=465506]: Done uploading piece. err: <nil> Dec 27 18:16:07.615121675 (530h47m7.48620472s) [noise=true , address=145.79.227.4:4185 , piece_id=2YKY3G2AUAQDRNWVISS2HCAXK4QEL33MXPPRR6FN3NJZV6ZBX2LQ, seg_pos={0 0}, upload=465506]: Done uploading piece. err: <nil> Dec 27 18:16:07.616187795 (530h47m7.487270831s) [noise=true , address=145.79.227.5:4223 , piece_id=H7W3YYWUVSTHAJUKWRK74ADPKMKE4AFU6YUNXHHUIY5NZVILXCZA, seg_pos={0 0}, upload=465506]: Done uploading piece. err: <nil> Dec 27 18:16:07.629081978 (530h47m7.500165023s) [noise=true , address=145.79.227.16:4911 , piece_id=EMTR4GXW55JY4E2RZ67AAF3IVAZZ2AR5T43BWCNCQ77XL5LRTW2A, seg_pos={0 0}, upload=465506]: Done uploading piece. err: <nil> Dec 27 18:16:07.630899488 (530h47m7.501982534s) [noise=true , address=145.79.227.25:4619 , piece_id=KOMO72DG6NKMRJJCWK77R3EVK2MVP5CAGEQ4QV54DVTAVIKSDXYQ, seg_pos={0 0}, upload=465506]: Done uploading piece. err: <nil> Dec 27 18:16:07.631569654 (530h47m7.502652699s) [noise=true , address=145.79.227.42:4123 , piece_id=6EQ5GKA3LAAHTW6KBS665BCVAT5DPU7LE7PEP2DEGO77AYOC5WHQ, seg_pos={0 0}, upload=465506]: Done uploading piece. err: <nil> Dec 27 18:16:07.791626823 (530h47m7.662709858s) [noise=true , address=77.90.31.17:4773 , piece_id=LJCIHM3FZNFVU4O4ZV4MWC6SKNGVDCYBXAVU7XD3C64ZLJGN3N3Q, seg_pos={0 0}, upload=465506]: Done uploading piece. err: <nil> Dec 27 18:16:07.803088648 (530h47m7.674171693s) [noise=true , address=77.90.31.10:4517 , piece_id=5UJOKHD6ZGY4XQWYAKTFHRBQZ6LHUWHFGQJZGLPHLGUQFTGLUTLA, seg_pos={0 0}, upload=465506]: Done uploading piece. err: <nil> Dec 27 18:16:07.863893813 (530h47m7.734976848s) [noise=true , address=45.91.116.5:4299 , piece_id=DE3PLUNUBWJD6RE5AARM5SDPWP3U32MFFBEDPR2USHXAUCYPXI5A, seg_pos={0 0}, upload=465506]: Done uploading piece. err: <nil> Dec 27 18:16:07.863919822 (530h47m7.735002868s) [seg_pos={0 0}, upload=465506]: Segment reached optimal threshold of 13 pieces. Dec 27 18:16:07.864162735 (530h47m7.73524579s) [noise=true , address=45.91.116.13:4153 , piece_id=RLOQLYRD5PCJ63NJZQ4OOZWMNJUPJTLY6GKQG4IMEKM75YGIA7HA, seg_pos={0 0}, upload=465506]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: write tcp 188.166.225.170:32792->45.91.116.13:4153: use of closed network connection; write tcp 188.166.225.170:32792->45.91.116.13:4153: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:32792->45.91.116.13:4153: use of closed network connection Dec 27 18:16:07.864173585 (530h47m7.735256661s) [noise=true , address=45.91.116.13:4153 , piece_id=RLOQLYRD5PCJ63NJZQ4OOZWMNJUPJTLY6GKQG4IMEKM75YGIA7HA, seg_pos={0 0}, upload=465506]: Long tail cancellation due to optimal threshold Dec 27 18:16:07.864395297 (530h47m7.735478352s) [noise=true , address=145.79.227.33:4869 , piece_id=KZRFP5EDEEFZYN5DWERCEGHIHLAXZDVNM6E4EF77ZZ6LIQTOWYEQ, seg_pos={0 0}, upload=465506]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18CQxjq65r6xVTmNRurSQPzKxj4BoR6cJ8Uq29sTrdh6uXm5sq): protocol: expected piece hash; context canceled; EOF Dec 27 18:16:07.864428390 (530h47m7.735511445s) [noise=true , address=145.79.227.33:4869 , piece_id=KZRFP5EDEEFZYN5DWERCEGHIHLAXZDVNM6E4EF77ZZ6LIQTOWYEQ, seg_pos={0 0}, upload=465506]: Long tail cancellation due to optimal threshold Dec 27 18:16:07.864582033 (530h47m7.735665078s) [noise=true , address=45.91.116.11:4120 , piece_id=Z5LI3ELXXRSIFZ33TOYWNXK6KFI6LB33SWZVBDIEQZA7WJKZQMAQ, seg_pos={0 0}, upload=465506]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: expected piece hash; context canceled; EOF Dec 27 18:16:07.864592393 (530h47m7.735675458s) [noise=true , address=45.91.116.11:4120 , piece_id=Z5LI3ELXXRSIFZ33TOYWNXK6KFI6LB33SWZVBDIEQZA7WJKZQMAQ, seg_pos={0 0}, upload=465506]: Long tail cancellation due to optimal threshold Dec 27 18:16:07.864621108 (530h47m7.735704153s) [seg_pos={0 0}, upload=465506]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 18:16:07.864716680 (530h47m7.735799725s) [upload=465506]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 18:16:07.864723713 (530h47m7.735806758s) [upload=465506]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 18:16:08.160809013 (530h47m8.031892058s) [upload=465506]: Done uploading segments... Dec 27 18:16:08.160830273 (530h47m8.031913308s) [upload=465506]: Upload finished. err: <nil> Duration: 30.004281306s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [................X...........................................] duration: 3.580819726s trace: https://jaeger.dev.storj.io/trace/3bddfef0dfa96557 FAIL (2025-12-27 18:16:19 UTC) orbiter warning: context deadline exceeded Duration: 3.580848781s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [.X.X.X..XXXX.X....XXX..XX....X...X.X........X.......X..X.XXX] duration: 30.003465693s trace: https://jaeger.dev.storj.io/trace/62eebb3fa7bb4444 FAIL (2025-12-27 18:15:59 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks, Piece ID: USL3JMPWANZDRPOBSWECYE2BCB7BFY3IQZPO2VZBQFAZHYBF3FMA) context deadline exceeded; piecestore: (Node ID: 12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, Piece ID: 6ANM4YJOYQBVSYHKR7MOZNQWSQCPJHVRDLHV2IBHIMRXLED7USHQ) context deadline exceeded; piecestore: (Node ID: 14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, Piece ID: PRRHOOBWPLY3HNTNONSQRZQO7LZUOMUH23WYMXUEQGJN2X42MIRQ) context deadline exceeded; piecestore: (Node ID: 1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, Piece ID: MQ62SQ2ZJJWSVK45IBEIOUVJ6SMJTEV4S5LBHC5CFS2DQENM2QZQ) context deadline exceeded ======LOGS====== 5 (530h46m29.84069601s) [upload=465499]: Done starting upload Dec 27 18:15:29.969674562 (530h46m29.840757598s) [upload=465499]: Uploading segments... Dec 27 18:15:29.969911033 (530h46m29.840994078s) [upload=465499]: Got next segment. Inline: false Dec 27 18:15:29.969961779 (530h46m29.841044824s) [upload=465499]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 18:15:29.969968371 (530h46m29.841051417s) [upload=465499]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 18:15:30.213936822 (530h46m30.085019857s) [seg_pos={0 0}, upload=465499]: Begin upload segment... Dec 27 18:15:30.214005783 (530h46m30.085088819s) [seg_pos={0 0}, upload=465499]: Done begin upload segment. Dec 27 18:15:30.214049116 (530h46m30.085132151s) [noise=true , address=145.79.227.48:4302 , piece_id=SDWGVGGIGVUJG5QAFBBAH7G56WIEQKN6ZZURSPJPISESA33GO34Q, seg_pos={0 0}, upload=465499]: Uploading piece... Dec 27 18:15:30.214224991 (530h46m30.085308026s) [noise=true , address=145.79.227.7:4295 , piece_id=B44OXY5HHCHBXO6W2HGOOQTEBIOA5T3FQSGE5ETJHACGJRWFRDHQ, seg_pos={0 0}, upload=465499]: Uploading piece... Dec 27 18:15:30.214707079 (530h46m30.085790114s) [noise=true , address=145.79.227.25:4618 , piece_id=SYQTFNYWS5KTDGTCLLP6GF5QSFBOIGX2FFG7UAIAYRXB5AB7YJJA, seg_pos={0 0}, upload=465499]: Uploading piece... Dec 27 18:15:30.214717759 (530h46m30.085800804s) [noise=true , address=145.79.227.41:4086 , piece_id=UWWGOL7GBV5P5JWQHQU4BOCX4FRYZ3I5WWP4W72FVLDVAU7OPVQQ, seg_pos={0 0}, upload=465499]: Uploading piece... Dec 27 18:15:30.215069288 (530h46m30.086152323s) [noise=true , address=45.91.116.4:4940 , piece_id=AIP3R2LBG7LQ5K7UL7NYQGXA2TVHPGDKNHHUYQ2IJMFH5XMYRJPQ, seg_pos={0 0}, upload=465499]: Uploading piece... Dec 27 18:15:30.215100928 (530h46m30.086183993s) [noise=true , address=77.90.31.18:4805 , piece_id=6ANM4YJOYQBVSYHKR7MOZNQWSQCPJHVRDLHV2IBHIMRXLED7USHQ, seg_pos={0 0}, upload=465499]: Uploading piece... Dec 27 18:15:30.215409075 (530h46m30.08649212s) [noise=true , address=145.79.227.51:4410 , piece_id=M3RHFT3C7ZPJIN4D4R7NKLVGLNW7UMLQ5275MRB6V23T5KALMICA, seg_pos={0 0}, upload=465499]: Uploading piece... Dec 27 18:15:30.215451816 (530h46m30.086534851s) [noise=true , address=77.90.31.16:4735 , piece_id=PRRHOOBWPLY3HNTNONSQRZQO7LZUOMUH23WYMXUEQGJN2X42MIRQ, seg_pos={0 0}, upload=465499]: Uploading piece... Dec 27 18:15:30.215711111 (530h46m30.086794146s) [noise=true , address=45.91.116.10:4082 , piece_id=MQ62SQ2ZJJWSVK45IBEIOUVJ6SMJTEV4S5LBHC5CFS2DQENM2QZQ, seg_pos={0 0}, upload=465499]: Uploading piece... Dec 27 18:15:30.215785202 (530h46m30.086868237s) [noise=true , address=45.91.116.6:4973 , piece_id=FVGIYNX3LRUHLOQQUSQZJCGYB4YBHXMALK6WPEUWCLUIGDIA4SOA, seg_pos={0 0}, upload=465499]: Uploading piece... Dec 27 18:15:30.215975193 (530h46m30.087058228s) [noise=true , address=145.79.227.30:4762 , piece_id=3LADVONE63MPMXUSIFFFLV4IVKIYRVHFIFVOD5TYEV6MVBCG3AYA, seg_pos={0 0}, upload=465499]: Uploading piece... Dec 27 18:15:30.216137973 (530h46m30.087221018s) [noise=true , address=145.79.227.38:4979 , piece_id=ABSTDODLB7TPARN3KMJWUZAJBSY4A3MXF2T5LBDDDH5GXBH6MZCA, seg_pos={0 0}, upload=465499]: Uploading piece... Dec 27 18:15:30.216236541 (530h46m30.087319576s) [noise=true , address=145.79.227.12:4767 , piece_id=PFNI7ALXGAQ6STOHJUR62JED326OBRUL4KLVLI4PI4WUGMIC3TRQ, seg_pos={0 0}, upload=465499]: Uploading piece... Dec 27 18:15:30.216415291 (530h46m30.087498326s) [noise=true , address=145.79.227.21:4474 , piece_id=BY77R44FURZNFNV3YKYT4XNJTOSYBQOXAAZBXLG5RTH4MR5PQ55Q, seg_pos={0 0}, upload=465499]: Uploading piece... Dec 27 18:15:30.216553244 (530h46m30.087636279s) [noise=true , address=77.90.31.11:4556 , piece_id=USL3JMPWANZDRPOBSWECYE2BCB7BFY3IQZPO2VZBQFAZHYBF3FMA, seg_pos={0 0}, upload=465499]: Uploading piece... Dec 27 18:15:30.216795666 (530h46m30.087878711s) [noise=true , address=145.79.227.52:4445 , piece_id=VC6P2KBNSD4D3TX363JTTTPTPKY7GG2MUMZ3EOSZNCTSPJESIZZA, seg_pos={0 0}, upload=465499]: Uploading piece... Dec 27 18:15:31.923787379 (530h46m31.794870424s) [upload=465499]: Next returned nil segment Dec 27 18:15:31.923837425 (530h46m31.79492047s) [upload=465499]: Waiting for error group managing segments... Dec 27 18:15:32.205741582 (530h46m32.076824627s) [noise=true , address=145.79.227.41:4086 , piece_id=UWWGOL7GBV5P5JWQHQU4BOCX4FRYZ3I5WWP4W72FVLDVAU7OPVQQ, seg_pos={0 0}, upload=465499]: Done uploading piece. err: <nil> Dec 27 18:15:32.206089064 (530h46m32.077172099s) [noise=true , address=145.79.227.25:4618 , piece_id=SYQTFNYWS5KTDGTCLLP6GF5QSFBOIGX2FFG7UAIAYRXB5AB7YJJA, seg_pos={0 0}, upload=465499]: Done uploading piece. err: <nil> Dec 27 18:15:32.219951650 (530h46m32.091034685s) [noise=true , address=145.79.227.7:4295 , piece_id=B44OXY5HHCHBXO6W2HGOOQTEBIOA5T3FQSGE5ETJHACGJRWFRDHQ, seg_pos={0 0}, upload=465499]: Done uploading piece. err: <nil> Dec 27 18:15:32.225069085 (530h46m32.09615213s) [noise=true , address=145.79.227.51:4410 , piece_id=M3RHFT3C7ZPJIN4D4R7NKLVGLNW7UMLQ5275MRB6V23T5KALMICA, seg_pos={0 0}, upload=465499]: Done uploading piece. err: <nil> Dec 27 18:15:32.236712436 (530h46m32.107795481s) [noise=true , address=145.79.227.48:4302 , piece_id=SDWGVGGIGVUJG5QAFBBAH7G56WIEQKN6ZZURSPJPISESA33GO34Q, seg_pos={0 0}, upload=465499]: Done uploading piece. err: <nil> Dec 27 18:15:32.236937805 (530h46m32.10802085s) [noise=true , address=145.79.227.38:4979 , piece_id=ABSTDODLB7TPARN3KMJWUZAJBSY4A3MXF2T5LBDDDH5GXBH6MZCA, seg_pos={0 0}, upload=465499]: Done uploading piece. err: <nil> Dec 27 18:15:32.242082001 (530h46m32.113165036s) [noise=true , address=145.79.227.12:4767 , piece_id=PFNI7ALXGAQ6STOHJUR62JED326OBRUL4KLVLI4PI4WUGMIC3TRQ, seg_pos={0 0}, upload=465499]: Done uploading piece. err: <nil> Dec 27 18:15:32.258198369 (530h46m32.129281404s) [noise=true , address=145.79.227.21:4474 , piece_id=BY77R44FURZNFNV3YKYT4XNJTOSYBQOXAAZBXLG5RTH4MR5PQ55Q, seg_pos={0 0}, upload=465499]: Done uploading piece. err: <nil> Dec 27 18:15:32.268042664 (530h46m32.139125699s) [noise=true , address=145.79.227.52:4445 , piece_id=VC6P2KBNSD4D3TX363JTTTPTPKY7GG2MUMZ3EOSZNCTSPJESIZZA, seg_pos={0 0}, upload=465499]: Done uploading piece. err: <nil> Dec 27 18:15:32.387293951 (530h46m32.258376996s) [noise=true , address=77.90.31.18:4805 , piece_id=6ANM4YJOYQBVSYHKR7MOZNQWSQCPJHVRDLHV2IBHIMRXLED7USHQ, seg_pos={0 0}, upload=465499]: Done uploading piece. err: <nil> Dec 27 18:15:32.438742588 (530h46m32.309825633s) [noise=true , address=77.90.31.11:4556 , piece_id=USL3JMPWANZDRPOBSWECYE2BCB7BFY3IQZPO2VZBQFAZHYBF3FMA, seg_pos={0 0}, upload=465499]: Done uploading piece. err: <nil> Dec 27 18:15:32.465667879 (530h46m32.336750914s) [noise=true , address=45.91.116.10:4082 , piece_id=MQ62SQ2ZJJWSVK45IBEIOUVJ6SMJTEV4S5LBHC5CFS2DQENM2QZQ, seg_pos={0 0}, upload=465499]: Done uploading piece. err: <nil> Dec 27 18:15:32.518857222 (530h46m32.389940267s) [noise=true , address=77.90.31.16:4735 , piece_id=PRRHOOBWPLY3HNTNONSQRZQO7LZUOMUH23WYMXUEQGJN2X42MIRQ, seg_pos={0 0}, upload=465499]: Done uploading piece. err: <nil> Dec 27 18:15:32.518879123 (530h46m32.389962168s) [seg_pos={0 0}, upload=465499]: Segment reached optimal threshold of 13 pieces. Dec 27 18:15:32.519155629 (530h46m32.390238674s) [noise=true , address=45.91.116.6:4973 , piece_id=FVGIYNX3LRUHLOQQUSQZJCGYB4YBHXMALK6WPEUWCLUIGDIA4SOA, seg_pos={0 0}, upload=465499]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV): protocol: read tcp 188.166.225.170:53650->45.91.116.6:4973: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 27 18:15:32.519162001 (530h46m32.390245046s) [noise=true , address=45.91.116.6:4973 , piece_id=FVGIYNX3LRUHLOQQUSQZJCGYB4YBHXMALK6WPEUWCLUIGDIA4SOA, seg_pos={0 0}, upload=465499]: Long tail cancellation due to optimal threshold Dec 27 18:15:32.519269375 (530h46m32.390352411s) [noise=true , address=45.91.116.4:4940 , piece_id=AIP3R2LBG7LQ5K7UL7NYQGXA2TVHPGDKNHHUYQ2IJMFH5XMYRJPQ, seg_pos={0 0}, upload=465499]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: write tcp 188.166.225.170:58276->45.91.116.4:4940: use of closed network connection; write tcp 188.166.225.170:58276->45.91.116.4:4940: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58276->45.91.116.4:4940: use of closed network connection Dec 27 18:15:32.519296818 (530h46m32.390379863s) [noise=true , address=45.91.116.4:4940 , piece_id=AIP3R2LBG7LQ5K7UL7NYQGXA2TVHPGDKNHHUYQ2IJMFH5XMYRJPQ, seg_pos={0 0}, upload=465499]: Long tail cancellation due to optimal threshold Dec 27 18:15:32.519394664 (530h46m32.39047771s) [noise=true , address=145.79.227.30:4762 , piece_id=3LADVONE63MPMXUSIFFFLV4IVKIYRVHFIFVOD5TYEV6MVBCG3AYA, seg_pos={0 0}, upload=465499]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b): protocol: expected piece hash; context canceled; EOF Dec 27 18:15:32.519402309 (530h46m32.390485354s) [noise=true , address=145.79.227.30:4762 , piece_id=3LADVONE63MPMXUSIFFFLV4IVKIYRVHFIFVOD5TYEV6MVBCG3AYA, seg_pos={0 0}, upload=465499]: Long tail cancellation due to optimal threshold Dec 27 18:15:32.519418680 (530h46m32.390501716s) [seg_pos={0 0}, upload=465499]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 18:15:32.519487091 (530h46m32.390570126s) [upload=465499]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 18:15:32.519492892 (530h46m32.390575927s) [upload=465499]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 18:15:32.825409242 (530h46m32.696492297s) [upload=465499]: Done uploading segments... Dec 27 18:15:32.825430032 (530h46m32.696513067s) [upload=465499]: Upload finished. err: <nil> Duration: 30.003487867s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [..X.X.X..XXXX.X....XXX..XX....X...X.X........X.......X..X.XX] duration: 30.00193396s trace: https://jaeger.dev.storj.io/trace/2f83678ac6713f2c FAIL (2025-12-27 18:15:22 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb, Piece ID: RGFYAIZYURNEYGQFFA7JF3I4HLLXVZFHGTMO5KNKOAMNGMTY6JHQ) context deadline exceeded; piecestore: (Node ID: 167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, Piece ID: QLZBG3EO53Y343BXMXQYGVMSPPQKPG2GPHD5L5BEID6WQZEGE25Q) context deadline exceeded; piecestore: (Node ID: 1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, Piece ID: URLQX3QW5Q3W3SQ7Z2EFOTFKZ2NSCSZISMMEECNX22XG2BMAM34Q) context deadline exceeded; piecestore: (Node ID: 1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj, Piece ID: 3T2OKWZSCLFB4WIE4LEKHWKN44OG5NCTWQ2QMJ37O35KAQ5BLWFQ) context deadline exceeded ======LOGS====== 7 18:14:52.893756667 (530h45m52.764839712s) [upload=465487]: Starting upload Dec 27 18:14:52.893971386 (530h45m52.765054441s) [upload=465487]: Done starting upload Dec 27 18:14:52.894039646 (530h45m52.765122691s) [upload=465487]: Uploading segments... Dec 27 18:14:52.894238394 (530h45m52.765321439s) [upload=465487]: Got next segment. Inline: false Dec 27 18:14:52.894287598 (530h45m52.765370643s) [upload=465487]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 18:14:52.894293429 (530h45m52.765376474s) [upload=465487]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 18:14:53.141156311 (530h45m53.012239346s) [seg_pos={0 0}, upload=465487]: Begin upload segment... Dec 27 18:14:53.141207768 (530h45m53.012290803s) [seg_pos={0 0}, upload=465487]: Done begin upload segment. Dec 27 18:14:53.141244408 (530h45m53.012327453s) [noise=true , address=45.91.116.11:4117 , piece_id=QLZBG3EO53Y343BXMXQYGVMSPPQKPG2GPHD5L5BEID6WQZEGE25Q, seg_pos={0 0}, upload=465487]: Uploading piece... Dec 27 18:14:53.141259046 (530h45m53.012342082s) [noise=true , address=145.79.227.50:4373 , piece_id=RHVLDFJ4JJQCFTGJXG7DRLWSETVWPJVI3FNP644U72FM2AFSTISQ, seg_pos={0 0}, upload=465487]: Uploading piece... Dec 27 18:14:53.141747867 (530h45m53.012831002s) [noise=true , address=145.79.227.11:4729 , piece_id=QPTTIKASRUVCHN2XGXAEXR36FOTKDBSLPFIUHVIEYS7GAWQIQXDQ, seg_pos={0 0}, upload=465487]: Uploading piece... Dec 27 18:14:53.142132780 (530h45m53.013215825s) [noise=true , address=77.90.31.15:4697 , piece_id=BX7WVPFEYDCRZPG7CM7O4FQA4EITAB6MA4UVDN47GBLQTH2YZCPQ, seg_pos={0 0}, upload=465487]: Uploading piece... Dec 27 18:14:53.142973661 (530h45m53.014056696s) [noise=true , address=145.79.227.46:4602 , piece_id=U275JNZFK45AQWZDXE6IWGK6IC6AMGTDQ557BRCHHC3PQX2LLHRQ, seg_pos={0 0}, upload=465487]: Uploading piece... Dec 27 18:14:53.143288631 (530h45m53.014371676s) [noise=true , address=145.79.227.5:4223 , piece_id=SJ6AUFGL55BM5L4Q475N5JPSTYLLVD2UNTZ2566HF2EZZS6VXE3A, seg_pos={0 0}, upload=465487]: Uploading piece... Dec 27 18:14:53.143569415 (530h45m53.01465245s) [noise=true , address=145.79.227.45:4229 , piece_id=N2SSVIYUKSC63Q2PSX5HDXFJATDYOLXH7H3HWITFMDUQDE646S5Q, seg_pos={0 0}, upload=465487]: Uploading piece... Dec 27 18:14:53.143873184 (530h45m53.014956219s) [noise=true , address=145.79.227.18:4007 , piece_id=WBY6JLUECRNXIRKOEPXP7ZCYYEVNQ6RYQV6ZRNEFFROGW5AZCBLQ, seg_pos={0 0}, upload=465487]: Uploading piece... Dec 27 18:14:53.144187994 (530h45m53.015271029s) [noise=true , address=45.91.116.14:4190 , piece_id=URLQX3QW5Q3W3SQ7Z2EFOTFKZ2NSCSZISMMEECNX22XG2BMAM34Q, seg_pos={0 0}, upload=465487]: Uploading piece... Dec 27 18:14:53.144528011 (530h45m53.015611046s) [noise=true , address=45.91.116.6:4976 , piece_id=UWPRC5JIFXZ6VTD7YJHUGE2UG2WC34LY462OERSRKSYYOWTPEZYQ, seg_pos={0 0}, upload=465487]: Uploading piece... Dec 27 18:14:53.145607316 (530h45m53.016690361s) [noise=true , address=45.91.116.10:4084 , piece_id=3T2OKWZSCLFB4WIE4LEKHWKN44OG5NCTWQ2QMJ37O35KAQ5BLWFQ, seg_pos={0 0}, upload=465487]: Uploading piece... Dec 27 18:14:53.145761560 (530h45m53.016844605s) [noise=true , address=145.79.227.17:4972 , piece_id=ACDOCH2YJRRG6PL2QDF5LGQPB6WM7ZNRHQI32LDVQWWJBF3OUMEA, seg_pos={0 0}, upload=465487]: Uploading piece... Dec 27 18:14:53.146998776 (530h45m53.018081811s) [noise=true , address=145.79.227.4:4186 , piece_id=MV5A34AZFQU247AFNAGK3P4S3WF4XQMMNLRFEBH352MCD7ZAWD6Q, seg_pos={0 0}, upload=465487]: Uploading piece... Dec 27 18:14:53.147562599 (530h45m53.018645645s) [noise=true , address=77.90.31.21:4880 , piece_id=RGFYAIZYURNEYGQFFA7JF3I4HLLXVZFHGTMO5KNKOAMNGMTY6JHQ, seg_pos={0 0}, upload=465487]: Uploading piece... Dec 27 18:14:53.148298901 (530h45m53.019381946s) [noise=true , address=45.91.116.8:4047 , piece_id=DFP4OYOCW24ZW3HWHAF7DDXAVCQMRY42JWMZBFXUDOYTS3MWV7BQ, seg_pos={0 0}, upload=465487]: Uploading piece... Dec 27 18:14:53.149764722 (530h45m53.020847757s) [noise=true , address=145.79.227.32:4834 , piece_id=DLRHMC5QLFINCRFZJSO5O73HBO2XDHBSY6IHS74XUJARKZ5NNDAQ, seg_pos={0 0}, upload=465487]: Uploading piece... Dec 27 18:14:54.829511888 (530h45m54.700594933s) [upload=465487]: Next returned nil segment Dec 27 18:14:54.829531675 (530h45m54.70061471s) [upload=465487]: Waiting for error group managing segments... Dec 27 18:14:55.156434197 (530h45m55.027517232s) [noise=true , address=145.79.227.17:4972 , piece_id=ACDOCH2YJRRG6PL2QDF5LGQPB6WM7ZNRHQI32LDVQWWJBF3OUMEA, seg_pos={0 0}, upload=465487]: Done uploading piece. err: <nil> Dec 27 18:14:55.168736434 (530h45m55.039819479s) [noise=true , address=145.79.227.46:4602 , piece_id=U275JNZFK45AQWZDXE6IWGK6IC6AMGTDQ557BRCHHC3PQX2LLHRQ, seg_pos={0 0}, upload=465487]: Done uploading piece. err: <nil> Dec 27 18:14:55.169404846 (530h45m55.040487891s) [noise=true , address=145.79.227.18:4007 , piece_id=WBY6JLUECRNXIRKOEPXP7ZCYYEVNQ6RYQV6ZRNEFFROGW5AZCBLQ, seg_pos={0 0}, upload=465487]: Done uploading piece. err: <nil> Dec 27 18:14:55.173130821 (530h45m55.044213856s) [noise=true , address=145.79.227.11:4729 , piece_id=QPTTIKASRUVCHN2XGXAEXR36FOTKDBSLPFIUHVIEYS7GAWQIQXDQ, seg_pos={0 0}, upload=465487]: Done uploading piece. err: <nil> Dec 27 18:14:55.174104174 (530h45m55.045187209s) [noise=true , address=145.79.227.5:4223 , piece_id=SJ6AUFGL55BM5L4Q475N5JPSTYLLVD2UNTZ2566HF2EZZS6VXE3A, seg_pos={0 0}, upload=465487]: Done uploading piece. err: <nil> Dec 27 18:14:55.175095082 (530h45m55.046178127s) [noise=true , address=145.79.227.45:4229 , piece_id=N2SSVIYUKSC63Q2PSX5HDXFJATDYOLXH7H3HWITFMDUQDE646S5Q, seg_pos={0 0}, upload=465487]: Done uploading piece. err: <nil> Dec 27 18:14:55.180467031 (530h45m55.051550076s) [noise=true , address=145.79.227.32:4834 , piece_id=DLRHMC5QLFINCRFZJSO5O73HBO2XDHBSY6IHS74XUJARKZ5NNDAQ, seg_pos={0 0}, upload=465487]: Done uploading piece. err: <nil> Dec 27 18:14:55.184060463 (530h45m55.055143498s) [noise=true , address=145.79.227.4:4186 , piece_id=MV5A34AZFQU247AFNAGK3P4S3WF4XQMMNLRFEBH352MCD7ZAWD6Q, seg_pos={0 0}, upload=465487]: Done uploading piece. err: <nil> Dec 27 18:14:55.188873498 (530h45m55.059956534s) [noise=true , address=145.79.227.50:4373 , piece_id=RHVLDFJ4JJQCFTGJXG7DRLWSETVWPJVI3FNP644U72FM2AFSTISQ, seg_pos={0 0}, upload=465487]: Done uploading piece. err: <nil> Dec 27 18:14:55.325657790 (530h45m55.196740825s) [noise=true , address=77.90.31.21:4880 , piece_id=RGFYAIZYURNEYGQFFA7JF3I4HLLXVZFHGTMO5KNKOAMNGMTY6JHQ, seg_pos={0 0}, upload=465487]: Done uploading piece. err: <nil> Dec 27 18:14:55.372051993 (530h45m55.243135028s) [noise=true , address=45.91.116.14:4190 , piece_id=URLQX3QW5Q3W3SQ7Z2EFOTFKZ2NSCSZISMMEECNX22XG2BMAM34Q, seg_pos={0 0}, upload=465487]: Done uploading piece. err: <nil> Dec 27 18:14:55.389520676 (530h45m55.260603711s) [noise=true , address=45.91.116.10:4084 , piece_id=3T2OKWZSCLFB4WIE4LEKHWKN44OG5NCTWQ2QMJ37O35KAQ5BLWFQ, seg_pos={0 0}, upload=465487]: Done uploading piece. err: <nil> Dec 27 18:14:55.414613028 (530h45m55.285696063s) [noise=true , address=45.91.116.11:4117 , piece_id=QLZBG3EO53Y343BXMXQYGVMSPPQKPG2GPHD5L5BEID6WQZEGE25Q, seg_pos={0 0}, upload=465487]: Done uploading piece. err: <nil> Dec 27 18:14:55.414631222 (530h45m55.285714257s) [seg_pos={0 0}, upload=465487]: Segment reached optimal threshold of 13 pieces. Dec 27 18:14:55.414816444 (530h45m55.285899489s) [noise=true , address=45.91.116.8:4047 , piece_id=DFP4OYOCW24ZW3HWHAF7DDXAVCQMRY42JWMZBFXUDOYTS3MWV7BQ, seg_pos={0 0}, upload=465487]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: write tcp 188.166.225.170:48288->45.91.116.8:4047: use of closed network connection; write tcp 188.166.225.170:48288->45.91.116.8:4047: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48288->45.91.116.8:4047: use of closed network connection Dec 27 18:14:55.414822998 (530h45m55.285906033s) [noise=true , address=45.91.116.8:4047 , piece_id=DFP4OYOCW24ZW3HWHAF7DDXAVCQMRY42JWMZBFXUDOYTS3MWV7BQ, seg_pos={0 0}, upload=465487]: Long tail cancellation due to optimal threshold Dec 27 18:14:55.415060529 (530h45m55.286143564s) [noise=true , address=45.91.116.6:4976 , piece_id=UWPRC5JIFXZ6VTD7YJHUGE2UG2WC34LY462OERSRKSYYOWTPEZYQ, seg_pos={0 0}, upload=465487]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp): protocol: expected piece hash; context canceled; EOF Dec 27 18:14:55.415066371 (530h45m55.286149416s) [noise=true , address=45.91.116.6:4976 , piece_id=UWPRC5JIFXZ6VTD7YJHUGE2UG2WC34LY462OERSRKSYYOWTPEZYQ, seg_pos={0 0}, upload=465487]: Long tail cancellation due to optimal threshold Dec 27 18:14:55.415189114 (530h45m55.286272149s) [noise=true , address=77.90.31.15:4697 , piece_id=BX7WVPFEYDCRZPG7CM7O4FQA4EITAB6MA4UVDN47GBLQTH2YZCPQ, seg_pos={0 0}, upload=465487]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5): protocol: expected piece hash; context canceled; EOF Dec 27 18:14:55.415195005 (530h45m55.28627805s) [noise=true , address=77.90.31.15:4697 , piece_id=BX7WVPFEYDCRZPG7CM7O4FQA4EITAB6MA4UVDN47GBLQTH2YZCPQ, seg_pos={0 0}, upload=465487]: Long tail cancellation due to optimal threshold Dec 27 18:14:55.415215414 (530h45m55.28629845s) [seg_pos={0 0}, upload=465487]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 18:14:55.415294044 (530h45m55.286377079s) [upload=465487]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 18:14:55.415299394 (530h45m55.286382439s) [upload=465487]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 18:14:55.702775913 (530h45m55.573858958s) [upload=465487]: Done uploading segments... Dec 27 18:14:55.702793426 (530h45m55.573876472s) [upload=465487]: Upload finished. err: <nil> Duration: 30.001955431s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.................X..........................................] duration: 3.570808822s trace: https://jaeger.dev.storj.io/trace/3bddfef0dfa96557 FAIL (2025-12-27 18:15:15 UTC) orbiter warning: context deadline exceeded Duration: 3.57083949s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..................X.........................................] duration: 3.600999651s trace: https://jaeger.dev.storj.io/trace/3bddfef0dfa96557 FAIL (2025-12-27 18:14:37 UTC) orbiter warning: context deadline exceeded Duration: 3.601018947s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [...X.X.X..XXXX.X....XXX..XX....X...X.X........X.......X..X.X] duration: 30.00269925s trace: https://jaeger.dev.storj.io/trace/51402a5fef03de8b FAIL (2025-12-27 18:14:17 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f, Piece ID: TGJ5COU6EJJG7ITPWYCWJLTBNBJDQK72KGUP3FX2N3UEXOJW7MQA) context deadline exceeded; piecestore: (Node ID: 1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, Piece ID: 3QH2WIOG6I6XGBNNG64LKJGNT7UN3SDJT3TQSGEDARVZ2JKVYANQ) context deadline exceeded ======LOGS====== 66 (530h44m47.609623711s) [upload=465469]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 18:13:47.979700357 (530h44m47.850783392s) [seg_pos={0 0}, upload=465469]: Begin upload segment... Dec 27 18:13:47.979767063 (530h44m47.850850108s) [seg_pos={0 0}, upload=465469]: Done begin upload segment. Dec 27 18:13:47.979833199 (530h44m47.850916234s) [noise=true , address=145.79.227.13:4804 , piece_id=RPT3CAVMLRDM24TD6Z4TR5SAV6WG3MSYOIRH3CAAD2K5GA5EKTCA, seg_pos={0 0}, upload=465469]: Uploading piece... Dec 27 18:13:47.979875150 (530h44m47.850958185s) [noise=true , address=45.91.116.13:4155 , piece_id=VS3DQPZ5YAAQV6AHAZHAJK47F2FPQD6EMONBPFS4D7N6IPMR6JLQ, seg_pos={0 0}, upload=465469]: Uploading piece... Dec 27 18:13:47.980650155 (530h44m47.85173319s) [noise=true , address=45.91.116.6:4975 , piece_id=5H7UWJMKFEAW5DSPE5RHXHXJF6JTGR2CXLF76WJ2NK2ITZWGR2RA, seg_pos={0 0}, upload=465469]: Uploading piece... Dec 27 18:13:47.980840517 (530h44m47.851923552s) [noise=true , address=45.91.116.10:4081 , piece_id=QINPZ7GBGVTR636G7ZMHFE7U4NRBYUAKLODIYQBVHPLQ4TOKBAZA, seg_pos={0 0}, upload=465469]: Uploading piece... Dec 27 18:13:47.981268132 (530h44m47.852351167s) [noise=true , address=145.79.227.20:4438 , piece_id=Y53JVF32LMI24VQHYVPJA7KLUYZ6ZZE54QDYF4Y3ST7572CC2VDQ, seg_pos={0 0}, upload=465469]: Uploading piece... Dec 27 18:13:47.981648666 (530h44m47.852731701s) [noise=true , address=45.91.116.14:4192 , piece_id=DAG2WI6P6W3LULPNEQQYRER2ZJZXGT4RGWVDWOHXFI3R2ODENQWQ, seg_pos={0 0}, upload=465469]: Uploading piece... Dec 27 18:13:47.982098262 (530h44m47.853181307s) [noise=true , address=145.79.227.41:4087 , piece_id=SDXF77W7MBK2VY47PV4TR5337ZOIGPFQKEUFGYAKGRWXJEK2L43Q, seg_pos={0 0}, upload=465469]: Uploading piece... Dec 27 18:13:47.982283185 (530h44m47.85336623s) [noise=true , address=45.91.116.9:4261 , piece_id=3QH2WIOG6I6XGBNNG64LKJGNT7UN3SDJT3TQSGEDARVZ2JKVYANQ, seg_pos={0 0}, upload=465469]: Uploading piece... Dec 27 18:13:47.982800330 (530h44m47.853883375s) [noise=true , address=77.90.31.13:4629 , piece_id=TGJ5COU6EJJG7ITPWYCWJLTBNBJDQK72KGUP3FX2N3UEXOJW7MQA, seg_pos={0 0}, upload=465469]: Uploading piece... Dec 27 18:13:47.982943322 (530h44m47.854026357s) [noise=true , address=145.79.227.12:4765 , piece_id=PWGLMZJYRJK47DHI2SQEKRFBAK3KJ4CZUTPOYW2LR3QYAKCI6HGQ, seg_pos={0 0}, upload=465469]: Uploading piece... Dec 27 18:13:47.983378350 (530h44m47.854461445s) [noise=true , address=145.79.227.50:4374 , piece_id=53JIC5DGNYRDV4RHOHX3THV56CAVXFQAS7JRJF4YRBBEX7ZXABQA, seg_pos={0 0}, upload=465469]: Uploading piece... Dec 27 18:13:47.983539477 (530h44m47.854622522s) [noise=true , address=145.79.227.42:4121 , piece_id=CP3FER6ZUG76XXIWUI4HNVSJVE6QOBR7H5NFP7FE4QL3LAOZAWPA, seg_pos={0 0}, upload=465469]: Uploading piece... Dec 27 18:13:47.983769194 (530h44m47.854852239s) [noise=true , address=45.91.116.4:4940 , piece_id=4G3UWPRYIJSI646FCCBAQFVHDPN6DC7NRA34FVXW4Z4HJ45L4U4Q, seg_pos={0 0}, upload=465469]: Uploading piece... Dec 27 18:13:47.984150039 (530h44m47.855233074s) [noise=true , address=145.79.227.26:4653 , piece_id=ID3ZTRVVZDQWGUKAXV57JIXYGBINYL6CHDOIKBIQZJM7T6E24CWQ, seg_pos={0 0}, upload=465469]: Uploading piece... Dec 27 18:13:47.987755845 (530h44m47.85883888s) [noise=true , address=145.79.227.11:4732 , piece_id=4Z2UBC36YZMGKE3HQUUVIYAD4XKIIDRULQC4B7ZKSCKR44IVTP6Q, seg_pos={0 0}, upload=465469]: Uploading piece... Dec 27 18:13:47.989967266 (530h44m47.861050311s) [noise=true , address=145.79.227.52:4447 , piece_id=ZOK5UHMVQUV6OG3YFBRU7LPRNAW6YPKGGB2ML7IMF2W3UTTAAZBA, seg_pos={0 0}, upload=465469]: Uploading piece... Dec 27 18:13:49.706990333 (530h44m49.578073378s) [upload=465469]: Next returned nil segment Dec 27 18:13:49.707017143 (530h44m49.578100188s) [upload=465469]: Waiting for error group managing segments... Dec 27 18:13:49.984803188 (530h44m49.855886234s) [noise=true , address=145.79.227.41:4087 , piece_id=SDXF77W7MBK2VY47PV4TR5337ZOIGPFQKEUFGYAKGRWXJEK2L43Q, seg_pos={0 0}, upload=465469]: Done uploading piece. err: <nil> Dec 27 18:13:49.988050882 (530h44m49.859133927s) [noise=true , address=145.79.227.20:4438 , piece_id=Y53JVF32LMI24VQHYVPJA7KLUYZ6ZZE54QDYF4Y3ST7572CC2VDQ, seg_pos={0 0}, upload=465469]: Done uploading piece. err: <nil> Dec 27 18:13:49.991294358 (530h44m49.862377403s) [noise=true , address=145.79.227.12:4765 , piece_id=PWGLMZJYRJK47DHI2SQEKRFBAK3KJ4CZUTPOYW2LR3QYAKCI6HGQ, seg_pos={0 0}, upload=465469]: Done uploading piece. err: <nil> Dec 27 18:13:49.991738845 (530h44m49.86282189s) [noise=true , address=145.79.227.13:4804 , piece_id=RPT3CAVMLRDM24TD6Z4TR5SAV6WG3MSYOIRH3CAAD2K5GA5EKTCA, seg_pos={0 0}, upload=465469]: Done uploading piece. err: <nil> Dec 27 18:13:49.992728981 (530h44m49.863812026s) [noise=true , address=145.79.227.11:4732 , piece_id=4Z2UBC36YZMGKE3HQUUVIYAD4XKIIDRULQC4B7ZKSCKR44IVTP6Q, seg_pos={0 0}, upload=465469]: Done uploading piece. err: <nil> Dec 27 18:13:49.996425469 (530h44m49.867508504s) [noise=true , address=145.79.227.52:4447 , piece_id=ZOK5UHMVQUV6OG3YFBRU7LPRNAW6YPKGGB2ML7IMF2W3UTTAAZBA, seg_pos={0 0}, upload=465469]: Done uploading piece. err: <nil> Dec 27 18:13:49.997475068 (530h44m49.868558113s) [noise=true , address=145.79.227.42:4121 , piece_id=CP3FER6ZUG76XXIWUI4HNVSJVE6QOBR7H5NFP7FE4QL3LAOZAWPA, seg_pos={0 0}, upload=465469]: Done uploading piece. err: <nil> Dec 27 18:13:49.998628654 (530h44m49.869711689s) [noise=true , address=145.79.227.50:4374 , piece_id=53JIC5DGNYRDV4RHOHX3THV56CAVXFQAS7JRJF4YRBBEX7ZXABQA, seg_pos={0 0}, upload=465469]: Done uploading piece. err: <nil> Dec 27 18:13:50.183418549 (530h44m50.054501584s) [noise=true , address=77.90.31.13:4629 , piece_id=TGJ5COU6EJJG7ITPWYCWJLTBNBJDQK72KGUP3FX2N3UEXOJW7MQA, seg_pos={0 0}, upload=465469]: Done uploading piece. err: <nil> Dec 27 18:13:50.249842521 (530h44m50.120925556s) [noise=true , address=45.91.116.10:4081 , piece_id=QINPZ7GBGVTR636G7ZMHFE7U4NRBYUAKLODIYQBVHPLQ4TOKBAZA, seg_pos={0 0}, upload=465469]: Done uploading piece. err: <nil> Dec 27 18:13:50.258882024 (530h44m50.12996507s) [noise=true , address=45.91.116.9:4261 , piece_id=3QH2WIOG6I6XGBNNG64LKJGNT7UN3SDJT3TQSGEDARVZ2JKVYANQ, seg_pos={0 0}, upload=465469]: Done uploading piece. err: <nil> Dec 27 18:13:50.341511886 (530h44m50.212594931s) [noise=true , address=45.91.116.14:4192 , piece_id=DAG2WI6P6W3LULPNEQQYRER2ZJZXGT4RGWVDWOHXFI3R2ODENQWQ, seg_pos={0 0}, upload=465469]: Done uploading piece. err: <nil> Dec 27 18:13:50.370300606 (530h44m50.241383651s) [noise=true , address=45.91.116.4:4940 , piece_id=4G3UWPRYIJSI646FCCBAQFVHDPN6DC7NRA34FVXW4Z4HJ45L4U4Q, seg_pos={0 0}, upload=465469]: Done uploading piece. err: <nil> Dec 27 18:13:50.370320934 (530h44m50.241403969s) [seg_pos={0 0}, upload=465469]: Segment reached optimal threshold of 13 pieces. Dec 27 18:13:50.370575830 (530h44m50.241658875s) [noise=true , address=45.91.116.13:4155 , piece_id=VS3DQPZ5YAAQV6AHAZHAJK47F2FPQD6EMONBPFS4D7N6IPMR6JLQ, seg_pos={0 0}, upload=465469]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf): protocol: write tcp 188.166.225.170:40456->45.91.116.13:4155: use of closed network connection; write tcp 188.166.225.170:40456->45.91.116.13:4155: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40456->45.91.116.13:4155: use of closed network connection Dec 27 18:13:50.370582332 (530h44m50.241665377s) [noise=true , address=45.91.116.13:4155 , piece_id=VS3DQPZ5YAAQV6AHAZHAJK47F2FPQD6EMONBPFS4D7N6IPMR6JLQ, seg_pos={0 0}, upload=465469]: Long tail cancellation due to optimal threshold Dec 27 18:13:50.370830244 (530h44m50.241913279s) [noise=true , address=145.79.227.26:4653 , piece_id=ID3ZTRVVZDQWGUKAXV57JIXYGBINYL6CHDOIKBIQZJM7T6E24CWQ, seg_pos={0 0}, upload=465469]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ): protocol: write tcp 188.166.225.170:60646->145.79.227.26:4653: use of closed network connection; write tcp 188.166.225.170:60646->145.79.227.26:4653: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60646->145.79.227.26:4653: use of closed network connection Dec 27 18:13:50.370837047 (530h44m50.241920082s) [noise=true , address=145.79.227.26:4653 , piece_id=ID3ZTRVVZDQWGUKAXV57JIXYGBINYL6CHDOIKBIQZJM7T6E24CWQ, seg_pos={0 0}, upload=465469]: Long tail cancellation due to optimal threshold Dec 27 18:13:50.370913943 (530h44m50.241996978s) [noise=true , address=45.91.116.6:4975 , piece_id=5H7UWJMKFEAW5DSPE5RHXHXJF6JTGR2CXLF76WJ2NK2ITZWGR2RA, seg_pos={0 0}, upload=465469]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: write tcp 188.166.225.170:56642->45.91.116.6:4975: use of closed network connection; write tcp 188.166.225.170:56642->45.91.116.6:4975: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56642->45.91.116.6:4975: use of closed network connection Dec 27 18:13:50.370918772 (530h44m50.242001807s) [noise=true , address=45.91.116.6:4975 , piece_id=5H7UWJMKFEAW5DSPE5RHXHXJF6JTGR2CXLF76WJ2NK2ITZWGR2RA, seg_pos={0 0}, upload=465469]: Long tail cancellation due to optimal threshold Dec 27 18:13:50.370937068 (530h44m50.242020113s) [seg_pos={0 0}, upload=465469]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 18:13:50.371002112 (530h44m50.242085147s) [upload=465469]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 18:13:50.371006820 (530h44m50.242089866s) [upload=465469]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 18:13:50.663362729 (530h44m50.534445774s) [upload=465469]: Done uploading segments... Dec 27 18:13:50.663425148 (530h44m50.534508193s) [upload=465469]: Upload finished. err: <nil> Duration:* 30.00273535s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...................X........................................] duration: 3.667845598s trace: https://jaeger.dev.storj.io/trace/3bddfef0dfa96557 FAIL (2025-12-27 18:14:10 UTC) orbiter warning: context deadline exceeded Duration: 3.667885974s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [....................X.......................................] duration: 3.676419384s trace: https://jaeger.dev.storj.io/trace/3bddfef0dfa96557 FAIL (2025-12-27 18:13:42 UTC) orbiter warning: context deadline exceeded Duration: 3.676447969s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.....................X......................................] duration: 3.696317004s trace: https://jaeger.dev.storj.io/trace/3bddfef0dfa96557 FAIL (2025-12-27 18:13:16 UTC) orbiter warning: context deadline exceeded Duration: 3.696341862s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [....X.X.X..XXXX.X....XXX..XX....X...X.X........X.......X..X.] duration: 5.751652537s trace: https://jaeger.dev.storj.io/trace/6a6c119fc9f2f9e7 FAIL (2025-12-27 18:13:05 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, Piece ID: 3KPXO3425PW5RDQBVS5F54IUUAAKMDLMHU563CCSAPRHWHCUFVPA) context deadline exceeded; piecestore: (Node ID: 12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, Piece ID: 37FH6YS6B6K5IHSTOV2ISL5WU6YTCULVYBMFBFOOQC7G6KNLNHJQ) context deadline exceeded ======LOGS====== Dec 27 18:12:30.253130849 (530h43m30.124213884s) [upload=465454]: Starting upload Dec 27 18:12:30.253190332 (530h43m30.124273467s) [upload=465454]: Done starting upload Dec 27 18:12:30.253372148 (530h43m30.124455183s) [upload=465454]: Uploading segments... Dec 27 18:12:30.253409830 (530h43m30.124492865s) [upload=465454]: Got next segment. Inline: false Dec 27 18:12:30.253439997 (530h43m30.124523032s) [upload=465454]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 18:12:30.253444376 (530h43m30.124527411s) [upload=465454]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 18:12:30.501322472 (530h43m30.372405507s) [seg_pos={0 0}, upload=465454]: Begin upload segment... Dec 27 18:12:30.501362297 (530h43m30.372445332s) [seg_pos={0 0}, upload=465454]: Done begin upload segment. Dec 27 18:12:30.501387105 (530h43m30.37247014s) [noise=true , address=45.91.116.5:4299 , piece_id=37FH6YS6B6K5IHSTOV2ISL5WU6YTCULVYBMFBFOOQC7G6KNLNHJQ, seg_pos={0 0}, upload=465454]: Uploading piece... Dec 27 18:12:30.501431810 (530h43m30.372514845s) [noise=true , address=45.91.116.12:4227 , piece_id=KCQUSKGFPWEMAKRXDH6277UO6MGR66L5D6BJSI2ABVJSFTVZBFUQ, seg_pos={0 0}, upload=465454]: Uploading piece... Dec 27 18:12:30.502013037 (530h43m30.373096072s) [noise=true , address=45.91.116.8:4047 , piece_id=QDBW64HCHUSVQU2SFGHCXFWLGECA4KGIVMVBDTE3ZQFVHLP4MZ5Q, seg_pos={0 0}, upload=465454]: Uploading piece... Dec 27 18:12:30.502389063 (530h43m30.373472098s) [noise=true , address=145.79.227.13:4804 , piece_id=ZHKWAWS7L2YBVY4GW4J3JSO4OOGDDGXX56LC4QJT5NXOQ4IWPWOQ, seg_pos={0 0}, upload=465454]: Uploading piece... Dec 27 18:12:30.502802079 (530h43m30.373885114s) [noise=true , address=145.79.227.17:4972 , piece_id=JVYHNYNWKO2H42ULEYKW2UGHHXJYH6BGT6OLHRLQYPND2PLXK2HQ, seg_pos={0 0}, upload=465454]: Uploading piece... Dec 27 18:12:30.503085228 (530h43m30.374168274s) [noise=true , address=145.79.227.12:4767 , piece_id=5JHYBVJHLGUSAC2OB3EXFCR3T64FXZDLTTM4EHBVIPTIQRY5XELQ, seg_pos={0 0}, upload=465454]: Uploading piece... Dec 27 18:12:30.503094826 (530h43m30.374177862s) [noise=true , address=145.79.227.38:4979 , piece_id=ZGQGFRD55AUZHURWS5CAK5ZDLVI7D2PXF5UYR36OEWTZ4KPHFD5A, seg_pos={0 0}, upload=465454]: Uploading piece... Dec 27 18:12:30.503363809 (530h43m30.374446844s) [noise=true , address=145.79.227.28:4727 , piece_id=AFR7DPV2I6KLG4HBCGGJJUJZWD5RXYWTBIEEK63BBUZ56ZQGJ4KQ, seg_pos={0 0}, upload=465454]: Uploading piece... Dec 27 18:12:30.503441066 (530h43m30.374524101s) [noise=true , address=77.90.31.13:4627 , piece_id=OV34VG3WGVVG5MC56PWFX63HAF3RYG5BRFYZUVR57SZF2XJHKFEQ, seg_pos={0 0}, upload=465454]: Uploading piece... Dec 27 18:12:30.503656897 (530h43m30.374739932s) [noise=true , address=145.79.227.46:4603 , piece_id=ZM43O3NIOK43WYECU66NGOH7WH4ETK77SDFDQ4YMFQSZC22AIVAQ, seg_pos={0 0}, upload=465454]: Uploading piece... Dec 27 18:12:30.503734564 (530h43m30.374817599s) [noise=true , address=145.79.227.4:4186 , piece_id=KJXMK5QRRBVDZYXMDFD7T5Z6DLA7F7LZOHHDHUYSGZY4LUPXS44Q, seg_pos={0 0}, upload=465454]: Uploading piece... Dec 27 18:12:30.504087457 (530h43m30.375170492s) [noise=true , address=45.91.116.11:4117 , piece_id=3KPXO3425PW5RDQBVS5F54IUUAAKMDLMHU563CCSAPRHWHCUFVPA, seg_pos={0 0}, upload=465454]: Uploading piece... Dec 27 18:12:30.504418527 (530h43m30.375501562s) [noise=true , address=45.91.116.10:4081 , piece_id=TDOOSQRIAL6CC7L22OYL6VTLFT5NQ6ZLAMQ2APTDALTUAEVPZWVQ, seg_pos={0 0}, upload=465454]: Uploading piece... Dec 27 18:12:30.504860830 (530h43m30.375943875s) [noise=true , address=145.79.227.50:4373 , piece_id=BCNKCQ6JAATZLCAWP3P3N46AQWVORGRHHQ4PIO76UUEIOHRITARQ, seg_pos={0 0}, upload=465454]: Uploading piece... Dec 27 18:12:30.505205145 (530h43m30.37628819s) [noise=true , address=145.79.227.41:4086 , piece_id=TYRKI6OKJAEFCEFPVVOGKB5OB6ZSMEPGHMV5BNQGLV7D2FLIKJZQ, seg_pos={0 0}, upload=465454]: Uploading piece... Dec 27 18:12:30.505810257 (530h43m30.376893302s) [noise=true , address=145.79.227.10:4402 , piece_id=YZFYCVFYUPFRYSKILXA5TNKCWSTE5FM6GZ5TC2472PTY5GAX26RQ, seg_pos={0 0}, upload=465454]: Uploading piece... Dec 27 18:12:32.218741461 (530h43m32.089824506s) [upload=465454]: Next returned nil segment Dec 27 18:12:32.218770316 (530h43m32.089853351s) [upload=465454]: Waiting for error group managing segments... Dec 27 18:12:32.492986534 (530h43m32.364069569s) [noise=true , address=145.79.227.12:4767 , piece_id=5JHYBVJHLGUSAC2OB3EXFCR3T64FXZDLTTM4EHBVIPTIQRY5XELQ, seg_pos={0 0}, upload=465454]: Done uploading piece. err: <nil> Dec 27 18:12:32.504134321 (530h43m32.375217356s) [noise=true , address=145.79.227.4:4186 , piece_id=KJXMK5QRRBVDZYXMDFD7T5Z6DLA7F7LZOHHDHUYSGZY4LUPXS44Q, seg_pos={0 0}, upload=465454]: Done uploading piece. err: <nil> Dec 27 18:12:32.505704661 (530h43m32.376787696s) [noise=true , address=145.79.227.50:4373 , piece_id=BCNKCQ6JAATZLCAWP3P3N46AQWVORGRHHQ4PIO76UUEIOHRITARQ, seg_pos={0 0}, upload=465454]: Done uploading piece. err: <nil> Dec 27 18:12:32.507556827 (530h43m32.378639872s) [noise=true , address=145.79.227.46:4603 , piece_id=ZM43O3NIOK43WYECU66NGOH7WH4ETK77SDFDQ4YMFQSZC22AIVAQ, seg_pos={0 0}, upload=465454]: Done uploading piece. err: <nil> Dec 27 18:12:32.511380208 (530h43m32.382463243s) [noise=true , address=145.79.227.13:4804 , piece_id=ZHKWAWS7L2YBVY4GW4J3JSO4OOGDDGXX56LC4QJT5NXOQ4IWPWOQ, seg_pos={0 0}, upload=465454]: Done uploading piece. err: <nil> Dec 27 18:12:32.512328814 (530h43m32.383411879s) [noise=true , address=145.79.227.10:4402 , piece_id=YZFYCVFYUPFRYSKILXA5TNKCWSTE5FM6GZ5TC2472PTY5GAX26RQ, seg_pos={0 0}, upload=465454]: Done uploading piece. err: <nil> Dec 27 18:12:32.515083861 (530h43m32.386166906s) [noise=true , address=145.79.227.17:4972 , piece_id=JVYHNYNWKO2H42ULEYKW2UGHHXJYH6BGT6OLHRLQYPND2PLXK2HQ, seg_pos={0 0}, upload=465454]: Done uploading piece. err: <nil> Dec 27 18:12:32.521216679 (530h43m32.392299714s) [noise=true , address=145.79.227.28:4727 , piece_id=AFR7DPV2I6KLG4HBCGGJJUJZWD5RXYWTBIEEK63BBUZ56ZQGJ4KQ, seg_pos={0 0}, upload=465454]: Done uploading piece. err: <nil> Dec 27 18:12:32.524792216 (530h43m32.395875262s) [noise=true , address=145.79.227.41:4086 , piece_id=TYRKI6OKJAEFCEFPVVOGKB5OB6ZSMEPGHMV5BNQGLV7D2FLIKJZQ, seg_pos={0 0}, upload=465454]: Done uploading piece. err: <nil> Dec 27 18:12:32.545149041 (530h43m32.416232076s) [noise=true , address=145.79.227.38:4979 , piece_id=ZGQGFRD55AUZHURWS5CAK5ZDLVI7D2PXF5UYR36OEWTZ4KPHFD5A, seg_pos={0 0}, upload=465454]: Done uploading piece. err: <nil> Dec 27 18:12:32.828283971 (530h43m32.699367006s) [noise=true , address=45.91.116.5:4299 , piece_id=37FH6YS6B6K5IHSTOV2ISL5WU6YTCULVYBMFBFOOQC7G6KNLNHJQ, seg_pos={0 0}, upload=465454]: Done uploading piece. err: <nil> Dec 27 18:12:32.852018135 (530h43m32.72310117s) [noise=true , address=45.91.116.11:4117 , piece_id=3KPXO3425PW5RDQBVS5F54IUUAAKMDLMHU563CCSAPRHWHCUFVPA, seg_pos={0 0}, upload=465454]: Done uploading piece. err: <nil> Dec 27 18:12:32.958203451 (530h43m32.829286496s) [noise=true , address=45.91.116.12:4227 , piece_id=KCQUSKGFPWEMAKRXDH6277UO6MGR66L5D6BJSI2ABVJSFTVZBFUQ, seg_pos={0 0}, upload=465454]: Done uploading piece. err: <nil> Dec 27 18:12:32.958222317 (530h43m32.829305352s) [seg_pos={0 0}, upload=465454]: Segment reached optimal threshold of 13 pieces. Dec 27 18:12:32.958407139 (530h43m32.829490184s) [noise=true , address=77.90.31.13:4627 , piece_id=OV34VG3WGVVG5MC56PWFX63HAF3RYG5BRFYZUVR57SZF2XJHKFEQ, seg_pos={0 0}, upload=465454]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB): protocol: expected piece hash; context canceled; EOF Dec 27 18:12:32.958413511 (530h43m32.829496546s) [noise=true , address=77.90.31.13:4627 , piece_id=OV34VG3WGVVG5MC56PWFX63HAF3RYG5BRFYZUVR57SZF2XJHKFEQ, seg_pos={0 0}, upload=465454]: Long tail cancellation due to optimal threshold Dec 27 18:12:32.958789096 (530h43m32.829872131s) [noise=true , address=45.91.116.10:4081 , piece_id=TDOOSQRIAL6CC7L22OYL6VTLFT5NQ6ZLAMQ2APTDALTUAEVPZWVQ, seg_pos={0 0}, upload=465454]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG): protocol: read tcp 188.166.225.170:44496->45.91.116.10:4081: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 27 18:12:32.958797021 (530h43m32.829880066s) [noise=true , address=45.91.116.10:4081 , piece_id=TDOOSQRIAL6CC7L22OYL6VTLFT5NQ6ZLAMQ2APTDALTUAEVPZWVQ, seg_pos={0 0}, upload=465454]: Long tail cancellation due to optimal threshold Dec 27 18:12:32.958961715 (530h43m32.83004476s) [noise=true , address=45.91.116.8:4047 , piece_id=QDBW64HCHUSVQU2SFGHCXFWLGECA4KGIVMVBDTE3ZQFVHLP4MZ5Q, seg_pos={0 0}, upload=465454]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: read tcp 188.166.225.170:45190->45.91.116.8:4047: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 27 18:12:32.958972325 (530h43m32.83005537s) [noise=true , address=45.91.116.8:4047 , piece_id=QDBW64HCHUSVQU2SFGHCXFWLGECA4KGIVMVBDTE3ZQFVHLP4MZ5Q, seg_pos={0 0}, upload=465454]: Long tail cancellation due to optimal threshold Dec 27 18:12:32.959012402 (530h43m32.830095437s) [seg_pos={0 0}, upload=465454]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 18:12:32.959127811 (530h43m32.830210856s) [upload=465454]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 18:12:32.959135005 (530h43m32.83021805s) [upload=465454]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 18:12:33.245092830 (530h43m33.116175875s) [upload=465454]: Done uploading segments... Dec 27 18:12:33.245114973 (530h43m33.116198018s) [upload=465454]: Upload finished. err: <nil> Duration: 5.751670082s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)