Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [X.X....X.......X.....................X......X.X.X...........] duration: 6.042138775s trace: https://jaeger.dev.storj.io/trace/69a6314bd3cfcdea FAIL (2025-12-29 20:31:02 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, Piece ID: MY4MH5QULSN3DBEGRFO75W2VHDANQNOHM2F6JNXIQUJXEGCL2J2A) context deadline exceeded; piecestore: (Node ID: 12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, Piece ID: 57WOZLBFS3KLYOTUM34EQ4RHBEOZEZP6BOWUMD2WK2QHDKQS257A) context deadline exceeded; piecestore: (Node ID: 1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, Piece ID: TLQONEXNWGMCRKW2AIZ25JK46DBEYPJGDIFYSQHIYEGHUDYCMG4A) context deadline exceeded ======LOGS====== Dec 29 20:23:28.415883495 (580h54m28.28696653s) [upload=509234]: Starting upload Dec 29 20:23:28.415951074 (580h54m28.287034109s) [upload=509234]: Done starting upload Dec 29 20:23:28.416064840 (580h54m28.287147885s) [upload=509234]: Uploading segments... Dec 29 20:23:28.416264120 (580h54m28.287347165s) [upload=509234]: Got next segment. Inline: false Dec 29 20:23:28.416310388 (580h54m28.287393423s) [upload=509234]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 20:23:28.416315598 (580h54m28.287398634s) [upload=509234]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 20:23:28.660976111 (580h54m28.532059146s) [seg_pos={0 0}, upload=509234]: Begin upload segment... Dec 29 20:23:28.661026877 (580h54m28.532109922s) [seg_pos={0 0}, upload=509234]: Done begin upload segment. Dec 29 20:23:28.661068466 (580h54m28.532151501s) [noise=true , address=145.79.227.48:4303 , piece_id=KC5NQZIPPNPLQKZL7EYL7RL5Q3HLL4GQQ6S42AHTDENJJ6ZX6ZJA, seg_pos={0 0}, upload=509234]: Uploading piece... Dec 29 20:23:28.661072965 (580h54m28.532156s) [noise=true , address=145.79.227.42:4123 , piece_id=OD5XMSISHKBNLSQNAMV3ZCLBFMRLM7CGS7XKPC2ZU2J5R5AD6GVQ, seg_pos={0 0}, upload=509234]: Uploading piece... Dec 29 20:23:28.661729705 (580h54m28.53281274s) [noise=true , address=145.79.227.39:4014 , piece_id=J2YF6RILHWJI7JLAWCYN4WXBRQMENM63WLRTKNV4H5S3ALX4Q5VQ, seg_pos={0 0}, upload=509234]: Uploading piece... Dec 29 20:23:28.662080493 (580h54m28.533163528s) [noise=true , address=45.91.116.8:4045 , piece_id=46CTNCF4EIA4QSWQRCGTFZRHDOAMFZG5VV3I2WBZYJE7YABX4HEQ, seg_pos={0 0}, upload=509234]: Uploading piece... Dec 29 20:23:28.662340749 (580h54m28.533423784s) [noise=true , address=77.90.31.11:4554 , piece_id=TLQONEXNWGMCRKW2AIZ25JK46DBEYPJGDIFYSQHIYEGHUDYCMG4A, seg_pos={0 0}, upload=509234]: Uploading piece... Dec 29 20:23:28.662600053 (580h54m28.533683098s) [noise=true , address=145.79.227.4:4187 , piece_id=FGSQSSM4EO73VDOYM5AJD2XZSYVU5LWUCOVOG3GRZGRCDUODSLDA, seg_pos={0 0}, upload=509234]: Uploading piece... Dec 29 20:23:28.664093767 (580h54m28.535176802s) [noise=true , address=145.79.227.47:4266 , piece_id=O3QWUUL63R42OY2CBZPPCIPLEMUTE7HJRYHHBOOEOUZU2NQZXXKQ, seg_pos={0 0}, upload=509234]: Uploading piece... Dec 29 20:23:28.664605071 (580h54m28.535688106s) [noise=true , address=145.79.227.35:4481 , piece_id=IUATLJFBJ5HG3H4W44EFZCRJ7U5V3HXJHBQRTMRGYWGAEVQXQX6Q, seg_pos={0 0}, upload=509234]: Uploading piece... Dec 29 20:23:28.665273063 (580h54m28.536356108s) [noise=true , address=77.90.31.21:4878 , piece_id=T6ZKRL26GO7CGZ23I4N56F66YHWUM6XJ6D3AI6VFICGAH6GNICJQ, seg_pos={0 0}, upload=509234]: Uploading piece... Dec 29 20:23:28.665755532 (580h54m28.536838577s) [noise=true , address=45.91.116.6:4974 , piece_id=MY4MH5QULSN3DBEGRFO75W2VHDANQNOHM2F6JNXIQUJXEGCL2J2A, seg_pos={0 0}, upload=509234]: Uploading piece... Dec 29 20:23:28.667333967 (580h54m28.538417012s) [noise=true , address=45.91.116.11:4119 , piece_id=OL6RAER4DLQDSIQFBPQACL36ZAY6YXFCTW3MVELQT44M6MW665LQ, seg_pos={0 0}, upload=509234]: Uploading piece... Dec 29 20:23:28.668065620 (580h54m28.539148665s) [noise=true , address=145.79.227.16:4912 , piece_id=MTCEEL6SYO6DXOXXRNDN243LSYSOFVBLTMMGPFJOBZCJDCEONEPA, seg_pos={0 0}, upload=509234]: Uploading piece... Dec 29 20:23:28.669481887 (580h54m28.540564922s) [noise=true , address=145.79.227.7:4293 , piece_id=LX36RRBBW2YRBQJDRQ4V3FA7TSNI5BVRZW7CCSYJNWDRTCRD5KTA, seg_pos={0 0}, upload=509234]: Uploading piece... Dec 29 20:23:28.670629553 (580h54m28.541712588s) [noise=true , address=145.79.227.41:4085 , piece_id=O26CLJH4ICQNMHBTENAJLRNLDFLHA3CV2RMNGK32IZPWEER6DY2Q, seg_pos={0 0}, upload=509234]: Uploading piece... Dec 29 20:23:28.671866527 (580h54m28.542949562s) [noise=true , address=45.91.116.10:4082 , piece_id=N6HSAF6XCLGUZXDNGBHCGB3CSMQMBNETCFW7CHU7LJGTZ4JJLREA, seg_pos={0 0}, upload=509234]: Uploading piece... Dec 29 20:23:28.672948639 (580h54m28.544031674s) [noise=true , address=45.91.116.12:4226 , piece_id=57WOZLBFS3KLYOTUM34EQ4RHBEOZEZP6BOWUMD2WK2QHDKQS257A, seg_pos={0 0}, upload=509234]: Uploading piece... Dec 29 20:23:30.355599227 (580h54m30.226682262s) [upload=509234]: Next returned nil segment Dec 29 20:23:30.355633262 (580h54m30.226716307s) [upload=509234]: Waiting for error group managing segments... Dec 29 20:23:30.642553302 (580h54m30.513636337s) [noise=true , address=145.79.227.42:4123 , piece_id=OD5XMSISHKBNLSQNAMV3ZCLBFMRLM7CGS7XKPC2ZU2J5R5AD6GVQ, seg_pos={0 0}, upload=509234]: Done uploading piece. err: <nil> Dec 29 20:23:30.647195511 (580h54m30.518278546s) [noise=true , address=145.79.227.4:4187 , piece_id=FGSQSSM4EO73VDOYM5AJD2XZSYVU5LWUCOVOG3GRZGRCDUODSLDA, seg_pos={0 0}, upload=509234]: Done uploading piece. err: <nil> Dec 29 20:23:30.661620258 (580h54m30.532703303s) [noise=true , address=145.79.227.48:4303 , piece_id=KC5NQZIPPNPLQKZL7EYL7RL5Q3HLL4GQQ6S42AHTDENJJ6ZX6ZJA, seg_pos={0 0}, upload=509234]: Done uploading piece. err: <nil> Dec 29 20:23:30.664794151 (580h54m30.535877196s) [noise=true , address=145.79.227.41:4085 , piece_id=O26CLJH4ICQNMHBTENAJLRNLDFLHA3CV2RMNGK32IZPWEER6DY2Q, seg_pos={0 0}, upload=509234]: Done uploading piece. err: <nil> Dec 29 20:23:30.677472743 (580h54m30.548555788s) [noise=true , address=145.79.227.7:4293 , piece_id=LX36RRBBW2YRBQJDRQ4V3FA7TSNI5BVRZW7CCSYJNWDRTCRD5KTA, seg_pos={0 0}, upload=509234]: Done uploading piece. err: <nil> Dec 29 20:23:30.679803722 (580h54m30.550886767s) [noise=true , address=145.79.227.16:4912 , piece_id=MTCEEL6SYO6DXOXXRNDN243LSYSOFVBLTMMGPFJOBZCJDCEONEPA, seg_pos={0 0}, upload=509234]: Done uploading piece. err: <nil> Dec 29 20:23:30.689244619 (580h54m30.560327654s) [noise=true , address=145.79.227.39:4014 , piece_id=J2YF6RILHWJI7JLAWCYN4WXBRQMENM63WLRTKNV4H5S3ALX4Q5VQ, seg_pos={0 0}, upload=509234]: Done uploading piece. err: <nil> Dec 29 20:23:30.700784082 (580h54m30.571867117s) [noise=true , address=145.79.227.47:4266 , piece_id=O3QWUUL63R42OY2CBZPPCIPLEMUTE7HJRYHHBOOEOUZU2NQZXXKQ, seg_pos={0 0}, upload=509234]: Done uploading piece. err: <nil> Dec 29 20:23:30.847297316 (580h54m30.718380351s) [noise=true , address=77.90.31.11:4554 , piece_id=TLQONEXNWGMCRKW2AIZ25JK46DBEYPJGDIFYSQHIYEGHUDYCMG4A, seg_pos={0 0}, upload=509234]: Done uploading piece. err: <nil> Dec 29 20:23:30.920591620 (580h54m30.791674655s) [noise=true , address=45.91.116.10:4082 , piece_id=N6HSAF6XCLGUZXDNGBHCGB3CSMQMBNETCFW7CHU7LJGTZ4JJLREA, seg_pos={0 0}, upload=509234]: Done uploading piece. err: <nil> Dec 29 20:23:30.926928366 (580h54m30.798011401s) [noise=true , address=45.91.116.12:4226 , piece_id=57WOZLBFS3KLYOTUM34EQ4RHBEOZEZP6BOWUMD2WK2QHDKQS257A, seg_pos={0 0}, upload=509234]: Done uploading piece. err: <nil> Dec 29 20:23:30.945981767 (580h54m30.817064812s) [noise=true , address=45.91.116.6:4974 , piece_id=MY4MH5QULSN3DBEGRFO75W2VHDANQNOHM2F6JNXIQUJXEGCL2J2A, seg_pos={0 0}, upload=509234]: Done uploading piece. err: <nil> Dec 29 20:23:30.958572291 (580h54m30.829655326s) [noise=true , address=77.90.31.21:4878 , piece_id=T6ZKRL26GO7CGZ23I4N56F66YHWUM6XJ6D3AI6VFICGAH6GNICJQ, seg_pos={0 0}, upload=509234]: Done uploading piece. err: <nil> Dec 29 20:23:30.958595034 (580h54m30.829678069s) [seg_pos={0 0}, upload=509234]: Segment reached optimal threshold of 13 pieces. Dec 29 20:23:30.959013401 (580h54m30.830096476s) [noise=true , address=145.79.227.35:4481 , piece_id=IUATLJFBJ5HG3H4W44EFZCRJ7U5V3HXJHBQRTMRGYWGAEVQXQX6Q, seg_pos={0 0}, upload=509234]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bRK8WWmAnWUwzCfHu8JtZgrruYSJCvhnsYTKcaLccJcv5RQjp): protocol: expected piece hash; context canceled; EOF Dec 29 20:23:30.959036535 (580h54m30.8301196s) [noise=true , address=145.79.227.35:4481 , piece_id=IUATLJFBJ5HG3H4W44EFZCRJ7U5V3HXJHBQRTMRGYWGAEVQXQX6Q, seg_pos={0 0}, upload=509234]: Long tail cancellation due to optimal threshold Dec 29 20:23:30.959062274 (580h54m30.830145309s) [noise=true , address=45.91.116.11:4119 , piece_id=OL6RAER4DLQDSIQFBPQACL36ZAY6YXFCTW3MVELQT44M6MW665LQ, seg_pos={0 0}, upload=509234]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm): protocol: expected piece hash; context canceled; EOF Dec 29 20:23:30.959070019 (580h54m30.830153054s) [noise=true , address=45.91.116.11:4119 , piece_id=OL6RAER4DLQDSIQFBPQACL36ZAY6YXFCTW3MVELQT44M6MW665LQ, seg_pos={0 0}, upload=509234]: Long tail cancellation due to optimal threshold Dec 29 20:23:30.959092252 (580h54m30.830175297s) [noise=true , address=45.91.116.8:4045 , piece_id=46CTNCF4EIA4QSWQRCGTFZRHDOAMFZG5VV3I2WBZYJE7YABX4HEQ, seg_pos={0 0}, upload=509234]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9): protocol: expected piece hash; context canceled; EOF Dec 29 20:23:30.959097441 (580h54m30.830180476s) [noise=true , address=45.91.116.8:4045 , piece_id=46CTNCF4EIA4QSWQRCGTFZRHDOAMFZG5VV3I2WBZYJE7YABX4HEQ, seg_pos={0 0}, upload=509234]: Long tail cancellation due to optimal threshold Dec 29 20:23:30.959117088 (580h54m30.830200123s) [seg_pos={0 0}, upload=509234]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 20:23:30.959186390 (580h54m30.830269425s) [upload=509234]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 20:23:30.959191369 (580h54m30.830274415s) [upload=509234]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 20:23:31.263824068 (580h54m31.134907103s) [upload=509234]: Done uploading segments... Dec 29 20:23:31.263840989 (580h54m31.134924024s) [upload=509234]: Upload finished. err: <nil> Duration: 6.042168011s core uplink us-select-1
checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing) antiblip: 1 out of 60 is failed (threshold=6) [............X...............................................] duration: 299.052672ms remote: "tcp:207.211.208.129:443" target: https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt trace: https://jaeger.dev.storj.io/trace/4679501fce62ca7 FAIL (2025-12-29 20:30:09 UTC) orbiter warning: Get "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt": context deadline exceeded Duration: 299.081488ms edge linksharing bgp eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [.X.X....X.......X.....................X......X.X.X..........] duration: 6.053992869s trace: https://jaeger.dev.storj.io/trace/69a6314bd3cfcdea FAIL (2025-12-29 20:30:00 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, Piece ID: MY4MH5QULSN3DBEGRFO75W2VHDANQNOHM2F6JNXIQUJXEGCL2J2A) context deadline exceeded; piecestore: (Node ID: 12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, Piece ID: 57WOZLBFS3KLYOTUM34EQ4RHBEOZEZP6BOWUMD2WK2QHDKQS257A) context deadline exceeded; piecestore: (Node ID: 1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, Piece ID: TLQONEXNWGMCRKW2AIZ25JK46DBEYPJGDIFYSQHIYEGHUDYCMG4A) context deadline exceeded ======LOGS====== Dec 29 20:23:28.415883495 (580h54m28.28696653s) [upload=509234]: Starting upload Dec 29 20:23:28.415951074 (580h54m28.287034109s) [upload=509234]: Done starting upload Dec 29 20:23:28.416064840 (580h54m28.287147885s) [upload=509234]: Uploading segments... Dec 29 20:23:28.416264120 (580h54m28.287347165s) [upload=509234]: Got next segment. Inline: false Dec 29 20:23:28.416310388 (580h54m28.287393423s) [upload=509234]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 20:23:28.416315598 (580h54m28.287398634s) [upload=509234]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 20:23:28.660976111 (580h54m28.532059146s) [seg_pos={0 0}, upload=509234]: Begin upload segment... Dec 29 20:23:28.661026877 (580h54m28.532109922s) [seg_pos={0 0}, upload=509234]: Done begin upload segment. Dec 29 20:23:28.661068466 (580h54m28.532151501s) [noise=true , address=145.79.227.48:4303 , piece_id=KC5NQZIPPNPLQKZL7EYL7RL5Q3HLL4GQQ6S42AHTDENJJ6ZX6ZJA, seg_pos={0 0}, upload=509234]: Uploading piece... Dec 29 20:23:28.661072965 (580h54m28.532156s) [noise=true , address=145.79.227.42:4123 , piece_id=OD5XMSISHKBNLSQNAMV3ZCLBFMRLM7CGS7XKPC2ZU2J5R5AD6GVQ, seg_pos={0 0}, upload=509234]: Uploading piece... Dec 29 20:23:28.661729705 (580h54m28.53281274s) [noise=true , address=145.79.227.39:4014 , piece_id=J2YF6RILHWJI7JLAWCYN4WXBRQMENM63WLRTKNV4H5S3ALX4Q5VQ, seg_pos={0 0}, upload=509234]: Uploading piece... Dec 29 20:23:28.662080493 (580h54m28.533163528s) [noise=true , address=45.91.116.8:4045 , piece_id=46CTNCF4EIA4QSWQRCGTFZRHDOAMFZG5VV3I2WBZYJE7YABX4HEQ, seg_pos={0 0}, upload=509234]: Uploading piece... Dec 29 20:23:28.662340749 (580h54m28.533423784s) [noise=true , address=77.90.31.11:4554 , piece_id=TLQONEXNWGMCRKW2AIZ25JK46DBEYPJGDIFYSQHIYEGHUDYCMG4A, seg_pos={0 0}, upload=509234]: Uploading piece... Dec 29 20:23:28.662600053 (580h54m28.533683098s) [noise=true , address=145.79.227.4:4187 , piece_id=FGSQSSM4EO73VDOYM5AJD2XZSYVU5LWUCOVOG3GRZGRCDUODSLDA, seg_pos={0 0}, upload=509234]: Uploading piece... Dec 29 20:23:28.664093767 (580h54m28.535176802s) [noise=true , address=145.79.227.47:4266 , piece_id=O3QWUUL63R42OY2CBZPPCIPLEMUTE7HJRYHHBOOEOUZU2NQZXXKQ, seg_pos={0 0}, upload=509234]: Uploading piece... Dec 29 20:23:28.664605071 (580h54m28.535688106s) [noise=true , address=145.79.227.35:4481 , piece_id=IUATLJFBJ5HG3H4W44EFZCRJ7U5V3HXJHBQRTMRGYWGAEVQXQX6Q, seg_pos={0 0}, upload=509234]: Uploading piece... Dec 29 20:23:28.665273063 (580h54m28.536356108s) [noise=true , address=77.90.31.21:4878 , piece_id=T6ZKRL26GO7CGZ23I4N56F66YHWUM6XJ6D3AI6VFICGAH6GNICJQ, seg_pos={0 0}, upload=509234]: Uploading piece... Dec 29 20:23:28.665755532 (580h54m28.536838577s) [noise=true , address=45.91.116.6:4974 , piece_id=MY4MH5QULSN3DBEGRFO75W2VHDANQNOHM2F6JNXIQUJXEGCL2J2A, seg_pos={0 0}, upload=509234]: Uploading piece... Dec 29 20:23:28.667333967 (580h54m28.538417012s) [noise=true , address=45.91.116.11:4119 , piece_id=OL6RAER4DLQDSIQFBPQACL36ZAY6YXFCTW3MVELQT44M6MW665LQ, seg_pos={0 0}, upload=509234]: Uploading piece... Dec 29 20:23:28.668065620 (580h54m28.539148665s) [noise=true , address=145.79.227.16:4912 , piece_id=MTCEEL6SYO6DXOXXRNDN243LSYSOFVBLTMMGPFJOBZCJDCEONEPA, seg_pos={0 0}, upload=509234]: Uploading piece... Dec 29 20:23:28.669481887 (580h54m28.540564922s) [noise=true , address=145.79.227.7:4293 , piece_id=LX36RRBBW2YRBQJDRQ4V3FA7TSNI5BVRZW7CCSYJNWDRTCRD5KTA, seg_pos={0 0}, upload=509234]: Uploading piece... Dec 29 20:23:28.670629553 (580h54m28.541712588s) [noise=true , address=145.79.227.41:4085 , piece_id=O26CLJH4ICQNMHBTENAJLRNLDFLHA3CV2RMNGK32IZPWEER6DY2Q, seg_pos={0 0}, upload=509234]: Uploading piece... Dec 29 20:23:28.671866527 (580h54m28.542949562s) [noise=true , address=45.91.116.10:4082 , piece_id=N6HSAF6XCLGUZXDNGBHCGB3CSMQMBNETCFW7CHU7LJGTZ4JJLREA, seg_pos={0 0}, upload=509234]: Uploading piece... Dec 29 20:23:28.672948639 (580h54m28.544031674s) [noise=true , address=45.91.116.12:4226 , piece_id=57WOZLBFS3KLYOTUM34EQ4RHBEOZEZP6BOWUMD2WK2QHDKQS257A, seg_pos={0 0}, upload=509234]: Uploading piece... Dec 29 20:23:30.355599227 (580h54m30.226682262s) [upload=509234]: Next returned nil segment Dec 29 20:23:30.355633262 (580h54m30.226716307s) [upload=509234]: Waiting for error group managing segments... Dec 29 20:23:30.642553302 (580h54m30.513636337s) [noise=true , address=145.79.227.42:4123 , piece_id=OD5XMSISHKBNLSQNAMV3ZCLBFMRLM7CGS7XKPC2ZU2J5R5AD6GVQ, seg_pos={0 0}, upload=509234]: Done uploading piece. err: <nil> Dec 29 20:23:30.647195511 (580h54m30.518278546s) [noise=true , address=145.79.227.4:4187 , piece_id=FGSQSSM4EO73VDOYM5AJD2XZSYVU5LWUCOVOG3GRZGRCDUODSLDA, seg_pos={0 0}, upload=509234]: Done uploading piece. err: <nil> Dec 29 20:23:30.661620258 (580h54m30.532703303s) [noise=true , address=145.79.227.48:4303 , piece_id=KC5NQZIPPNPLQKZL7EYL7RL5Q3HLL4GQQ6S42AHTDENJJ6ZX6ZJA, seg_pos={0 0}, upload=509234]: Done uploading piece. err: <nil> Dec 29 20:23:30.664794151 (580h54m30.535877196s) [noise=true , address=145.79.227.41:4085 , piece_id=O26CLJH4ICQNMHBTENAJLRNLDFLHA3CV2RMNGK32IZPWEER6DY2Q, seg_pos={0 0}, upload=509234]: Done uploading piece. err: <nil> Dec 29 20:23:30.677472743 (580h54m30.548555788s) [noise=true , address=145.79.227.7:4293 , piece_id=LX36RRBBW2YRBQJDRQ4V3FA7TSNI5BVRZW7CCSYJNWDRTCRD5KTA, seg_pos={0 0}, upload=509234]: Done uploading piece. err: <nil> Dec 29 20:23:30.679803722 (580h54m30.550886767s) [noise=true , address=145.79.227.16:4912 , piece_id=MTCEEL6SYO6DXOXXRNDN243LSYSOFVBLTMMGPFJOBZCJDCEONEPA, seg_pos={0 0}, upload=509234]: Done uploading piece. err: <nil> Dec 29 20:23:30.689244619 (580h54m30.560327654s) [noise=true , address=145.79.227.39:4014 , piece_id=J2YF6RILHWJI7JLAWCYN4WXBRQMENM63WLRTKNV4H5S3ALX4Q5VQ, seg_pos={0 0}, upload=509234]: Done uploading piece. err: <nil> Dec 29 20:23:30.700784082 (580h54m30.571867117s) [noise=true , address=145.79.227.47:4266 , piece_id=O3QWUUL63R42OY2CBZPPCIPLEMUTE7HJRYHHBOOEOUZU2NQZXXKQ, seg_pos={0 0}, upload=509234]: Done uploading piece. err: <nil> Dec 29 20:23:30.847297316 (580h54m30.718380351s) [noise=true , address=77.90.31.11:4554 , piece_id=TLQONEXNWGMCRKW2AIZ25JK46DBEYPJGDIFYSQHIYEGHUDYCMG4A, seg_pos={0 0}, upload=509234]: Done uploading piece. err: <nil> Dec 29 20:23:30.920591620 (580h54m30.791674655s) [noise=true , address=45.91.116.10:4082 , piece_id=N6HSAF6XCLGUZXDNGBHCGB3CSMQMBNETCFW7CHU7LJGTZ4JJLREA, seg_pos={0 0}, upload=509234]: Done uploading piece. err: <nil> Dec 29 20:23:30.926928366 (580h54m30.798011401s) [noise=true , address=45.91.116.12:4226 , piece_id=57WOZLBFS3KLYOTUM34EQ4RHBEOZEZP6BOWUMD2WK2QHDKQS257A, seg_pos={0 0}, upload=509234]: Done uploading piece. err: <nil> Dec 29 20:23:30.945981767 (580h54m30.817064812s) [noise=true , address=45.91.116.6:4974 , piece_id=MY4MH5QULSN3DBEGRFO75W2VHDANQNOHM2F6JNXIQUJXEGCL2J2A, seg_pos={0 0}, upload=509234]: Done uploading piece. err: <nil> Dec 29 20:23:30.958572291 (580h54m30.829655326s) [noise=true , address=77.90.31.21:4878 , piece_id=T6ZKRL26GO7CGZ23I4N56F66YHWUM6XJ6D3AI6VFICGAH6GNICJQ, seg_pos={0 0}, upload=509234]: Done uploading piece. err: <nil> Dec 29 20:23:30.958595034 (580h54m30.829678069s) [seg_pos={0 0}, upload=509234]: Segment reached optimal threshold of 13 pieces. Dec 29 20:23:30.959013401 (580h54m30.830096476s) [noise=true , address=145.79.227.35:4481 , piece_id=IUATLJFBJ5HG3H4W44EFZCRJ7U5V3HXJHBQRTMRGYWGAEVQXQX6Q, seg_pos={0 0}, upload=509234]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bRK8WWmAnWUwzCfHu8JtZgrruYSJCvhnsYTKcaLccJcv5RQjp): protocol: expected piece hash; context canceled; EOF Dec 29 20:23:30.959036535 (580h54m30.8301196s) [noise=true , address=145.79.227.35:4481 , piece_id=IUATLJFBJ5HG3H4W44EFZCRJ7U5V3HXJHBQRTMRGYWGAEVQXQX6Q, seg_pos={0 0}, upload=509234]: Long tail cancellation due to optimal threshold Dec 29 20:23:30.959062274 (580h54m30.830145309s) [noise=true , address=45.91.116.11:4119 , piece_id=OL6RAER4DLQDSIQFBPQACL36ZAY6YXFCTW3MVELQT44M6MW665LQ, seg_pos={0 0}, upload=509234]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm): protocol: expected piece hash; context canceled; EOF Dec 29 20:23:30.959070019 (580h54m30.830153054s) [noise=true , address=45.91.116.11:4119 , piece_id=OL6RAER4DLQDSIQFBPQACL36ZAY6YXFCTW3MVELQT44M6MW665LQ, seg_pos={0 0}, upload=509234]: Long tail cancellation due to optimal threshold Dec 29 20:23:30.959092252 (580h54m30.830175297s) [noise=true , address=45.91.116.8:4045 , piece_id=46CTNCF4EIA4QSWQRCGTFZRHDOAMFZG5VV3I2WBZYJE7YABX4HEQ, seg_pos={0 0}, upload=509234]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9): protocol: expected piece hash; context canceled; EOF Dec 29 20:23:30.959097441 (580h54m30.830180476s) [noise=true , address=45.91.116.8:4045 , piece_id=46CTNCF4EIA4QSWQRCGTFZRHDOAMFZG5VV3I2WBZYJE7YABX4HEQ, seg_pos={0 0}, upload=509234]: Long tail cancellation due to optimal threshold Dec 29 20:23:30.959117088 (580h54m30.830200123s) [seg_pos={0 0}, upload=509234]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 20:23:30.959186390 (580h54m30.830269425s) [upload=509234]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 20:23:30.959191369 (580h54m30.830274415s) [upload=509234]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 20:23:31.263824068 (580h54m31.134907103s) [upload=509234]: Done uploading segments... Dec 29 20:23:31.263840989 (580h54m31.134924024s) [upload=509234]: Upload finished. err: <nil> Duration: 6.054076009s core uplink us-select-1
checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing) antiblip: 1 out of 60 is failed (threshold=6) [.............X..............................................] duration: 251.012758ms remote: "tcp:207.211.208.129:443" target: https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt trace: https://jaeger.dev.storj.io/trace/4679501fce62ca7 FAIL (2025-12-29 20:29:53 UTC) orbiter warning: Get "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt": context deadline exceeded Duration: 251.091067ms edge linksharing bgp eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [..X.X....X.......X.....................X......X.X.X.........] duration: 6.248368904s trace: https://jaeger.dev.storj.io/trace/69a6314bd3cfcdea FAIL (2025-12-29 20:29:28 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, Piece ID: MY4MH5QULSN3DBEGRFO75W2VHDANQNOHM2F6JNXIQUJXEGCL2J2A) context deadline exceeded; piecestore: (Node ID: 12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, Piece ID: 57WOZLBFS3KLYOTUM34EQ4RHBEOZEZP6BOWUMD2WK2QHDKQS257A) context deadline exceeded; piecestore: (Node ID: 1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, Piece ID: TLQONEXNWGMCRKW2AIZ25JK46DBEYPJGDIFYSQHIYEGHUDYCMG4A) context deadline exceeded ======LOGS====== Dec 29 20:23:28.415883495 (580h54m28.28696653s) [upload=509234]: Starting upload Dec 29 20:23:28.415951074 (580h54m28.287034109s) [upload=509234]: Done starting upload Dec 29 20:23:28.416064840 (580h54m28.287147885s) [upload=509234]: Uploading segments... Dec 29 20:23:28.416264120 (580h54m28.287347165s) [upload=509234]: Got next segment. Inline: false Dec 29 20:23:28.416310388 (580h54m28.287393423s) [upload=509234]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 20:23:28.416315598 (580h54m28.287398634s) [upload=509234]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 20:23:28.660976111 (580h54m28.532059146s) [seg_pos={0 0}, upload=509234]: Begin upload segment... Dec 29 20:23:28.661026877 (580h54m28.532109922s) [seg_pos={0 0}, upload=509234]: Done begin upload segment. Dec 29 20:23:28.661068466 (580h54m28.532151501s) [noise=true , address=145.79.227.48:4303 , piece_id=KC5NQZIPPNPLQKZL7EYL7RL5Q3HLL4GQQ6S42AHTDENJJ6ZX6ZJA, seg_pos={0 0}, upload=509234]: Uploading piece... Dec 29 20:23:28.661072965 (580h54m28.532156s) [noise=true , address=145.79.227.42:4123 , piece_id=OD5XMSISHKBNLSQNAMV3ZCLBFMRLM7CGS7XKPC2ZU2J5R5AD6GVQ, seg_pos={0 0}, upload=509234]: Uploading piece... Dec 29 20:23:28.661729705 (580h54m28.53281274s) [noise=true , address=145.79.227.39:4014 , piece_id=J2YF6RILHWJI7JLAWCYN4WXBRQMENM63WLRTKNV4H5S3ALX4Q5VQ, seg_pos={0 0}, upload=509234]: Uploading piece... Dec 29 20:23:28.662080493 (580h54m28.533163528s) [noise=true , address=45.91.116.8:4045 , piece_id=46CTNCF4EIA4QSWQRCGTFZRHDOAMFZG5VV3I2WBZYJE7YABX4HEQ, seg_pos={0 0}, upload=509234]: Uploading piece... Dec 29 20:23:28.662340749 (580h54m28.533423784s) [noise=true , address=77.90.31.11:4554 , piece_id=TLQONEXNWGMCRKW2AIZ25JK46DBEYPJGDIFYSQHIYEGHUDYCMG4A, seg_pos={0 0}, upload=509234]: Uploading piece... Dec 29 20:23:28.662600053 (580h54m28.533683098s) [noise=true , address=145.79.227.4:4187 , piece_id=FGSQSSM4EO73VDOYM5AJD2XZSYVU5LWUCOVOG3GRZGRCDUODSLDA, seg_pos={0 0}, upload=509234]: Uploading piece... Dec 29 20:23:28.664093767 (580h54m28.535176802s) [noise=true , address=145.79.227.47:4266 , piece_id=O3QWUUL63R42OY2CBZPPCIPLEMUTE7HJRYHHBOOEOUZU2NQZXXKQ, seg_pos={0 0}, upload=509234]: Uploading piece... Dec 29 20:23:28.664605071 (580h54m28.535688106s) [noise=true , address=145.79.227.35:4481 , piece_id=IUATLJFBJ5HG3H4W44EFZCRJ7U5V3HXJHBQRTMRGYWGAEVQXQX6Q, seg_pos={0 0}, upload=509234]: Uploading piece... Dec 29 20:23:28.665273063 (580h54m28.536356108s) [noise=true , address=77.90.31.21:4878 , piece_id=T6ZKRL26GO7CGZ23I4N56F66YHWUM6XJ6D3AI6VFICGAH6GNICJQ, seg_pos={0 0}, upload=509234]: Uploading piece... Dec 29 20:23:28.665755532 (580h54m28.536838577s) [noise=true , address=45.91.116.6:4974 , piece_id=MY4MH5QULSN3DBEGRFO75W2VHDANQNOHM2F6JNXIQUJXEGCL2J2A, seg_pos={0 0}, upload=509234]: Uploading piece... Dec 29 20:23:28.667333967 (580h54m28.538417012s) [noise=true , address=45.91.116.11:4119 , piece_id=OL6RAER4DLQDSIQFBPQACL36ZAY6YXFCTW3MVELQT44M6MW665LQ, seg_pos={0 0}, upload=509234]: Uploading piece... Dec 29 20:23:28.668065620 (580h54m28.539148665s) [noise=true , address=145.79.227.16:4912 , piece_id=MTCEEL6SYO6DXOXXRNDN243LSYSOFVBLTMMGPFJOBZCJDCEONEPA, seg_pos={0 0}, upload=509234]: Uploading piece... Dec 29 20:23:28.669481887 (580h54m28.540564922s) [noise=true , address=145.79.227.7:4293 , piece_id=LX36RRBBW2YRBQJDRQ4V3FA7TSNI5BVRZW7CCSYJNWDRTCRD5KTA, seg_pos={0 0}, upload=509234]: Uploading piece... Dec 29 20:23:28.670629553 (580h54m28.541712588s) [noise=true , address=145.79.227.41:4085 , piece_id=O26CLJH4ICQNMHBTENAJLRNLDFLHA3CV2RMNGK32IZPWEER6DY2Q, seg_pos={0 0}, upload=509234]: Uploading piece... Dec 29 20:23:28.671866527 (580h54m28.542949562s) [noise=true , address=45.91.116.10:4082 , piece_id=N6HSAF6XCLGUZXDNGBHCGB3CSMQMBNETCFW7CHU7LJGTZ4JJLREA, seg_pos={0 0}, upload=509234]: Uploading piece... Dec 29 20:23:28.672948639 (580h54m28.544031674s) [noise=true , address=45.91.116.12:4226 , piece_id=57WOZLBFS3KLYOTUM34EQ4RHBEOZEZP6BOWUMD2WK2QHDKQS257A, seg_pos={0 0}, upload=509234]: Uploading piece... Dec 29 20:23:30.355599227 (580h54m30.226682262s) [upload=509234]: Next returned nil segment Dec 29 20:23:30.355633262 (580h54m30.226716307s) [upload=509234]: Waiting for error group managing segments... Dec 29 20:23:30.642553302 (580h54m30.513636337s) [noise=true , address=145.79.227.42:4123 , piece_id=OD5XMSISHKBNLSQNAMV3ZCLBFMRLM7CGS7XKPC2ZU2J5R5AD6GVQ, seg_pos={0 0}, upload=509234]: Done uploading piece. err: <nil> Dec 29 20:23:30.647195511 (580h54m30.518278546s) [noise=true , address=145.79.227.4:4187 , piece_id=FGSQSSM4EO73VDOYM5AJD2XZSYVU5LWUCOVOG3GRZGRCDUODSLDA, seg_pos={0 0}, upload=509234]: Done uploading piece. err: <nil> Dec 29 20:23:30.661620258 (580h54m30.532703303s) [noise=true , address=145.79.227.48:4303 , piece_id=KC5NQZIPPNPLQKZL7EYL7RL5Q3HLL4GQQ6S42AHTDENJJ6ZX6ZJA, seg_pos={0 0}, upload=509234]: Done uploading piece. err: <nil> Dec 29 20:23:30.664794151 (580h54m30.535877196s) [noise=true , address=145.79.227.41:4085 , piece_id=O26CLJH4ICQNMHBTENAJLRNLDFLHA3CV2RMNGK32IZPWEER6DY2Q, seg_pos={0 0}, upload=509234]: Done uploading piece. err: <nil> Dec 29 20:23:30.677472743 (580h54m30.548555788s) [noise=true , address=145.79.227.7:4293 , piece_id=LX36RRBBW2YRBQJDRQ4V3FA7TSNI5BVRZW7CCSYJNWDRTCRD5KTA, seg_pos={0 0}, upload=509234]: Done uploading piece. err: <nil> Dec 29 20:23:30.679803722 (580h54m30.550886767s) [noise=true , address=145.79.227.16:4912 , piece_id=MTCEEL6SYO6DXOXXRNDN243LSYSOFVBLTMMGPFJOBZCJDCEONEPA, seg_pos={0 0}, upload=509234]: Done uploading piece. err: <nil> Dec 29 20:23:30.689244619 (580h54m30.560327654s) [noise=true , address=145.79.227.39:4014 , piece_id=J2YF6RILHWJI7JLAWCYN4WXBRQMENM63WLRTKNV4H5S3ALX4Q5VQ, seg_pos={0 0}, upload=509234]: Done uploading piece. err: <nil> Dec 29 20:23:30.700784082 (580h54m30.571867117s) [noise=true , address=145.79.227.47:4266 , piece_id=O3QWUUL63R42OY2CBZPPCIPLEMUTE7HJRYHHBOOEOUZU2NQZXXKQ, seg_pos={0 0}, upload=509234]: Done uploading piece. err: <nil> Dec 29 20:23:30.847297316 (580h54m30.718380351s) [noise=true , address=77.90.31.11:4554 , piece_id=TLQONEXNWGMCRKW2AIZ25JK46DBEYPJGDIFYSQHIYEGHUDYCMG4A, seg_pos={0 0}, upload=509234]: Done uploading piece. err: <nil> Dec 29 20:23:30.920591620 (580h54m30.791674655s) [noise=true , address=45.91.116.10:4082 , piece_id=N6HSAF6XCLGUZXDNGBHCGB3CSMQMBNETCFW7CHU7LJGTZ4JJLREA, seg_pos={0 0}, upload=509234]: Done uploading piece. err: <nil> Dec 29 20:23:30.926928366 (580h54m30.798011401s) [noise=true , address=45.91.116.12:4226 , piece_id=57WOZLBFS3KLYOTUM34EQ4RHBEOZEZP6BOWUMD2WK2QHDKQS257A, seg_pos={0 0}, upload=509234]: Done uploading piece. err: <nil> Dec 29 20:23:30.945981767 (580h54m30.817064812s) [noise=true , address=45.91.116.6:4974 , piece_id=MY4MH5QULSN3DBEGRFO75W2VHDANQNOHM2F6JNXIQUJXEGCL2J2A, seg_pos={0 0}, upload=509234]: Done uploading piece. err: <nil> Dec 29 20:23:30.958572291 (580h54m30.829655326s) [noise=true , address=77.90.31.21:4878 , piece_id=T6ZKRL26GO7CGZ23I4N56F66YHWUM6XJ6D3AI6VFICGAH6GNICJQ, seg_pos={0 0}, upload=509234]: Done uploading piece. err: <nil> Dec 29 20:23:30.958595034 (580h54m30.829678069s) [seg_pos={0 0}, upload=509234]: Segment reached optimal threshold of 13 pieces. Dec 29 20:23:30.959013401 (580h54m30.830096476s) [noise=true , address=145.79.227.35:4481 , piece_id=IUATLJFBJ5HG3H4W44EFZCRJ7U5V3HXJHBQRTMRGYWGAEVQXQX6Q, seg_pos={0 0}, upload=509234]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bRK8WWmAnWUwzCfHu8JtZgrruYSJCvhnsYTKcaLccJcv5RQjp): protocol: expected piece hash; context canceled; EOF Dec 29 20:23:30.959036535 (580h54m30.8301196s) [noise=true , address=145.79.227.35:4481 , piece_id=IUATLJFBJ5HG3H4W44EFZCRJ7U5V3HXJHBQRTMRGYWGAEVQXQX6Q, seg_pos={0 0}, upload=509234]: Long tail cancellation due to optimal threshold Dec 29 20:23:30.959062274 (580h54m30.830145309s) [noise=true , address=45.91.116.11:4119 , piece_id=OL6RAER4DLQDSIQFBPQACL36ZAY6YXFCTW3MVELQT44M6MW665LQ, seg_pos={0 0}, upload=509234]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm): protocol: expected piece hash; context canceled; EOF Dec 29 20:23:30.959070019 (580h54m30.830153054s) [noise=true , address=45.91.116.11:4119 , piece_id=OL6RAER4DLQDSIQFBPQACL36ZAY6YXFCTW3MVELQT44M6MW665LQ, seg_pos={0 0}, upload=509234]: Long tail cancellation due to optimal threshold Dec 29 20:23:30.959092252 (580h54m30.830175297s) [noise=true , address=45.91.116.8:4045 , piece_id=46CTNCF4EIA4QSWQRCGTFZRHDOAMFZG5VV3I2WBZYJE7YABX4HEQ, seg_pos={0 0}, upload=509234]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9): protocol: expected piece hash; context canceled; EOF Dec 29 20:23:30.959097441 (580h54m30.830180476s) [noise=true , address=45.91.116.8:4045 , piece_id=46CTNCF4EIA4QSWQRCGTFZRHDOAMFZG5VV3I2WBZYJE7YABX4HEQ, seg_pos={0 0}, upload=509234]: Long tail cancellation due to optimal threshold Dec 29 20:23:30.959117088 (580h54m30.830200123s) [seg_pos={0 0}, upload=509234]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 20:23:30.959186390 (580h54m30.830269425s) [upload=509234]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 20:23:30.959191369 (580h54m30.830274415s) [upload=509234]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 20:23:31.263824068 (580h54m31.134907103s) [upload=509234]: Done uploading segments... Dec 29 20:23:31.263840989 (580h54m31.134924024s) [upload=509234]: Upload finished. err: <nil> Duration: 6.248384022s core uplink us-select-1
checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing) antiblip: 1 out of 60 is failed (threshold=6) [..............X.............................................] duration: 266.824576ms remote: "tcp:207.211.208.129:443" target: https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt trace: https://jaeger.dev.storj.io/trace/4679501fce62ca7 FAIL (2025-12-29 20:29:19 UTC) orbiter warning: Get "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt": context deadline exceeded Duration: 266.849102ms edge linksharing bgp eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [X..X.X....X.......X.....................X......X.X.X........] duration: 10.156304384s trace: https://jaeger.dev.storj.io/trace/69a6314bd3cfcdea FAIL (2025-12-29 20:29:14 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, Piece ID: MY4MH5QULSN3DBEGRFO75W2VHDANQNOHM2F6JNXIQUJXEGCL2J2A) context deadline exceeded; piecestore: (Node ID: 12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, Piece ID: 57WOZLBFS3KLYOTUM34EQ4RHBEOZEZP6BOWUMD2WK2QHDKQS257A) context deadline exceeded; piecestore: (Node ID: 1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, Piece ID: TLQONEXNWGMCRKW2AIZ25JK46DBEYPJGDIFYSQHIYEGHUDYCMG4A) context deadline exceeded ======LOGS====== Dec 29 20:23:28.415883495 (580h54m28.28696653s) [upload=509234]: Starting upload Dec 29 20:23:28.415951074 (580h54m28.287034109s) [upload=509234]: Done starting upload Dec 29 20:23:28.416064840 (580h54m28.287147885s) [upload=509234]: Uploading segments... Dec 29 20:23:28.416264120 (580h54m28.287347165s) [upload=509234]: Got next segment. Inline: false Dec 29 20:23:28.416310388 (580h54m28.287393423s) [upload=509234]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 20:23:28.416315598 (580h54m28.287398634s) [upload=509234]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 20:23:28.660976111 (580h54m28.532059146s) [seg_pos={0 0}, upload=509234]: Begin upload segment... Dec 29 20:23:28.661026877 (580h54m28.532109922s) [seg_pos={0 0}, upload=509234]: Done begin upload segment. Dec 29 20:23:28.661068466 (580h54m28.532151501s) [noise=true , address=145.79.227.48:4303 , piece_id=KC5NQZIPPNPLQKZL7EYL7RL5Q3HLL4GQQ6S42AHTDENJJ6ZX6ZJA, seg_pos={0 0}, upload=509234]: Uploading piece... Dec 29 20:23:28.661072965 (580h54m28.532156s) [noise=true , address=145.79.227.42:4123 , piece_id=OD5XMSISHKBNLSQNAMV3ZCLBFMRLM7CGS7XKPC2ZU2J5R5AD6GVQ, seg_pos={0 0}, upload=509234]: Uploading piece... Dec 29 20:23:28.661729705 (580h54m28.53281274s) [noise=true , address=145.79.227.39:4014 , piece_id=J2YF6RILHWJI7JLAWCYN4WXBRQMENM63WLRTKNV4H5S3ALX4Q5VQ, seg_pos={0 0}, upload=509234]: Uploading piece... Dec 29 20:23:28.662080493 (580h54m28.533163528s) [noise=true , address=45.91.116.8:4045 , piece_id=46CTNCF4EIA4QSWQRCGTFZRHDOAMFZG5VV3I2WBZYJE7YABX4HEQ, seg_pos={0 0}, upload=509234]: Uploading piece... Dec 29 20:23:28.662340749 (580h54m28.533423784s) [noise=true , address=77.90.31.11:4554 , piece_id=TLQONEXNWGMCRKW2AIZ25JK46DBEYPJGDIFYSQHIYEGHUDYCMG4A, seg_pos={0 0}, upload=509234]: Uploading piece... Dec 29 20:23:28.662600053 (580h54m28.533683098s) [noise=true , address=145.79.227.4:4187 , piece_id=FGSQSSM4EO73VDOYM5AJD2XZSYVU5LWUCOVOG3GRZGRCDUODSLDA, seg_pos={0 0}, upload=509234]: Uploading piece... Dec 29 20:23:28.664093767 (580h54m28.535176802s) [noise=true , address=145.79.227.47:4266 , piece_id=O3QWUUL63R42OY2CBZPPCIPLEMUTE7HJRYHHBOOEOUZU2NQZXXKQ, seg_pos={0 0}, upload=509234]: Uploading piece... Dec 29 20:23:28.664605071 (580h54m28.535688106s) [noise=true , address=145.79.227.35:4481 , piece_id=IUATLJFBJ5HG3H4W44EFZCRJ7U5V3HXJHBQRTMRGYWGAEVQXQX6Q, seg_pos={0 0}, upload=509234]: Uploading piece... Dec 29 20:23:28.665273063 (580h54m28.536356108s) [noise=true , address=77.90.31.21:4878 , piece_id=T6ZKRL26GO7CGZ23I4N56F66YHWUM6XJ6D3AI6VFICGAH6GNICJQ, seg_pos={0 0}, upload=509234]: Uploading piece... Dec 29 20:23:28.665755532 (580h54m28.536838577s) [noise=true , address=45.91.116.6:4974 , piece_id=MY4MH5QULSN3DBEGRFO75W2VHDANQNOHM2F6JNXIQUJXEGCL2J2A, seg_pos={0 0}, upload=509234]: Uploading piece... Dec 29 20:23:28.667333967 (580h54m28.538417012s) [noise=true , address=45.91.116.11:4119 , piece_id=OL6RAER4DLQDSIQFBPQACL36ZAY6YXFCTW3MVELQT44M6MW665LQ, seg_pos={0 0}, upload=509234]: Uploading piece... Dec 29 20:23:28.668065620 (580h54m28.539148665s) [noise=true , address=145.79.227.16:4912 , piece_id=MTCEEL6SYO6DXOXXRNDN243LSYSOFVBLTMMGPFJOBZCJDCEONEPA, seg_pos={0 0}, upload=509234]: Uploading piece... Dec 29 20:23:28.669481887 (580h54m28.540564922s) [noise=true , address=145.79.227.7:4293 , piece_id=LX36RRBBW2YRBQJDRQ4V3FA7TSNI5BVRZW7CCSYJNWDRTCRD5KTA, seg_pos={0 0}, upload=509234]: Uploading piece... Dec 29 20:23:28.670629553 (580h54m28.541712588s) [noise=true , address=145.79.227.41:4085 , piece_id=O26CLJH4ICQNMHBTENAJLRNLDFLHA3CV2RMNGK32IZPWEER6DY2Q, seg_pos={0 0}, upload=509234]: Uploading piece... Dec 29 20:23:28.671866527 (580h54m28.542949562s) [noise=true , address=45.91.116.10:4082 , piece_id=N6HSAF6XCLGUZXDNGBHCGB3CSMQMBNETCFW7CHU7LJGTZ4JJLREA, seg_pos={0 0}, upload=509234]: Uploading piece... Dec 29 20:23:28.672948639 (580h54m28.544031674s) [noise=true , address=45.91.116.12:4226 , piece_id=57WOZLBFS3KLYOTUM34EQ4RHBEOZEZP6BOWUMD2WK2QHDKQS257A, seg_pos={0 0}, upload=509234]: Uploading piece... Dec 29 20:23:30.355599227 (580h54m30.226682262s) [upload=509234]: Next returned nil segment Dec 29 20:23:30.355633262 (580h54m30.226716307s) [upload=509234]: Waiting for error group managing segments... Dec 29 20:23:30.642553302 (580h54m30.513636337s) [noise=true , address=145.79.227.42:4123 , piece_id=OD5XMSISHKBNLSQNAMV3ZCLBFMRLM7CGS7XKPC2ZU2J5R5AD6GVQ, seg_pos={0 0}, upload=509234]: Done uploading piece. err: <nil> Dec 29 20:23:30.647195511 (580h54m30.518278546s) [noise=true , address=145.79.227.4:4187 , piece_id=FGSQSSM4EO73VDOYM5AJD2XZSYVU5LWUCOVOG3GRZGRCDUODSLDA, seg_pos={0 0}, upload=509234]: Done uploading piece. err: <nil> Dec 29 20:23:30.661620258 (580h54m30.532703303s) [noise=true , address=145.79.227.48:4303 , piece_id=KC5NQZIPPNPLQKZL7EYL7RL5Q3HLL4GQQ6S42AHTDENJJ6ZX6ZJA, seg_pos={0 0}, upload=509234]: Done uploading piece. err: <nil> Dec 29 20:23:30.664794151 (580h54m30.535877196s) [noise=true , address=145.79.227.41:4085 , piece_id=O26CLJH4ICQNMHBTENAJLRNLDFLHA3CV2RMNGK32IZPWEER6DY2Q, seg_pos={0 0}, upload=509234]: Done uploading piece. err: <nil> Dec 29 20:23:30.677472743 (580h54m30.548555788s) [noise=true , address=145.79.227.7:4293 , piece_id=LX36RRBBW2YRBQJDRQ4V3FA7TSNI5BVRZW7CCSYJNWDRTCRD5KTA, seg_pos={0 0}, upload=509234]: Done uploading piece. err: <nil> Dec 29 20:23:30.679803722 (580h54m30.550886767s) [noise=true , address=145.79.227.16:4912 , piece_id=MTCEEL6SYO6DXOXXRNDN243LSYSOFVBLTMMGPFJOBZCJDCEONEPA, seg_pos={0 0}, upload=509234]: Done uploading piece. err: <nil> Dec 29 20:23:30.689244619 (580h54m30.560327654s) [noise=true , address=145.79.227.39:4014 , piece_id=J2YF6RILHWJI7JLAWCYN4WXBRQMENM63WLRTKNV4H5S3ALX4Q5VQ, seg_pos={0 0}, upload=509234]: Done uploading piece. err: <nil> Dec 29 20:23:30.700784082 (580h54m30.571867117s) [noise=true , address=145.79.227.47:4266 , piece_id=O3QWUUL63R42OY2CBZPPCIPLEMUTE7HJRYHHBOOEOUZU2NQZXXKQ, seg_pos={0 0}, upload=509234]: Done uploading piece. err: <nil> Dec 29 20:23:30.847297316 (580h54m30.718380351s) [noise=true , address=77.90.31.11:4554 , piece_id=TLQONEXNWGMCRKW2AIZ25JK46DBEYPJGDIFYSQHIYEGHUDYCMG4A, seg_pos={0 0}, upload=509234]: Done uploading piece. err: <nil> Dec 29 20:23:30.920591620 (580h54m30.791674655s) [noise=true , address=45.91.116.10:4082 , piece_id=N6HSAF6XCLGUZXDNGBHCGB3CSMQMBNETCFW7CHU7LJGTZ4JJLREA, seg_pos={0 0}, upload=509234]: Done uploading piece. err: <nil> Dec 29 20:23:30.926928366 (580h54m30.798011401s) [noise=true , address=45.91.116.12:4226 , piece_id=57WOZLBFS3KLYOTUM34EQ4RHBEOZEZP6BOWUMD2WK2QHDKQS257A, seg_pos={0 0}, upload=509234]: Done uploading piece. err: <nil> Dec 29 20:23:30.945981767 (580h54m30.817064812s) [noise=true , address=45.91.116.6:4974 , piece_id=MY4MH5QULSN3DBEGRFO75W2VHDANQNOHM2F6JNXIQUJXEGCL2J2A, seg_pos={0 0}, upload=509234]: Done uploading piece. err: <nil> Dec 29 20:23:30.958572291 (580h54m30.829655326s) [noise=true , address=77.90.31.21:4878 , piece_id=T6ZKRL26GO7CGZ23I4N56F66YHWUM6XJ6D3AI6VFICGAH6GNICJQ, seg_pos={0 0}, upload=509234]: Done uploading piece. err: <nil> Dec 29 20:23:30.958595034 (580h54m30.829678069s) [seg_pos={0 0}, upload=509234]: Segment reached optimal threshold of 13 pieces. Dec 29 20:23:30.959013401 (580h54m30.830096476s) [noise=true , address=145.79.227.35:4481 , piece_id=IUATLJFBJ5HG3H4W44EFZCRJ7U5V3HXJHBQRTMRGYWGAEVQXQX6Q, seg_pos={0 0}, upload=509234]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bRK8WWmAnWUwzCfHu8JtZgrruYSJCvhnsYTKcaLccJcv5RQjp): protocol: expected piece hash; context canceled; EOF Dec 29 20:23:30.959036535 (580h54m30.8301196s) [noise=true , address=145.79.227.35:4481 , piece_id=IUATLJFBJ5HG3H4W44EFZCRJ7U5V3HXJHBQRTMRGYWGAEVQXQX6Q, seg_pos={0 0}, upload=509234]: Long tail cancellation due to optimal threshold Dec 29 20:23:30.959062274 (580h54m30.830145309s) [noise=true , address=45.91.116.11:4119 , piece_id=OL6RAER4DLQDSIQFBPQACL36ZAY6YXFCTW3MVELQT44M6MW665LQ, seg_pos={0 0}, upload=509234]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm): protocol: expected piece hash; context canceled; EOF Dec 29 20:23:30.959070019 (580h54m30.830153054s) [noise=true , address=45.91.116.11:4119 , piece_id=OL6RAER4DLQDSIQFBPQACL36ZAY6YXFCTW3MVELQT44M6MW665LQ, seg_pos={0 0}, upload=509234]: Long tail cancellation due to optimal threshold Dec 29 20:23:30.959092252 (580h54m30.830175297s) [noise=true , address=45.91.116.8:4045 , piece_id=46CTNCF4EIA4QSWQRCGTFZRHDOAMFZG5VV3I2WBZYJE7YABX4HEQ, seg_pos={0 0}, upload=509234]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9): protocol: expected piece hash; context canceled; EOF Dec 29 20:23:30.959097441 (580h54m30.830180476s) [noise=true , address=45.91.116.8:4045 , piece_id=46CTNCF4EIA4QSWQRCGTFZRHDOAMFZG5VV3I2WBZYJE7YABX4HEQ, seg_pos={0 0}, upload=509234]: Long tail cancellation due to optimal threshold Dec 29 20:23:30.959117088 (580h54m30.830200123s) [seg_pos={0 0}, upload=509234]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 20:23:30.959186390 (580h54m30.830269425s) [upload=509234]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 20:23:30.959191369 (580h54m30.830274415s) [upload=509234]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 20:23:31.263824068 (580h54m31.134907103s) [upload=509234]: Done uploading segments... Dec 29 20:23:31.263840989 (580h54m31.134924024s) [upload=509234]: Upload finished. err: <nil> Duration: 10.156331875s core uplink us-select-1
checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing) antiblip: 1 out of 60 is failed (threshold=6) [...............X............................................] duration: 277.356609ms remote: "tcp:207.211.208.129:443" target: https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt trace: https://jaeger.dev.storj.io/trace/4679501fce62ca7 FAIL (2025-12-29 20:28:52 UTC) orbiter warning: Get "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt": context deadline exceeded Duration: 277.405133ms edge linksharing bgp eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [.X..X.X....X.......X.....................X......X.X.X.......] duration: 6.056723586s trace: https://jaeger.dev.storj.io/trace/69a6314bd3cfcdea FAIL (2025-12-29 20:28:10 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, Piece ID: MY4MH5QULSN3DBEGRFO75W2VHDANQNOHM2F6JNXIQUJXEGCL2J2A) context deadline exceeded; piecestore: (Node ID: 12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, Piece ID: 57WOZLBFS3KLYOTUM34EQ4RHBEOZEZP6BOWUMD2WK2QHDKQS257A) context deadline exceeded; piecestore: (Node ID: 1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, Piece ID: TLQONEXNWGMCRKW2AIZ25JK46DBEYPJGDIFYSQHIYEGHUDYCMG4A) context deadline exceeded ======LOGS====== Dec 29 20:23:28.415883495 (580h54m28.28696653s) [upload=509234]: Starting upload Dec 29 20:23:28.415951074 (580h54m28.287034109s) [upload=509234]: Done starting upload Dec 29 20:23:28.416064840 (580h54m28.287147885s) [upload=509234]: Uploading segments... Dec 29 20:23:28.416264120 (580h54m28.287347165s) [upload=509234]: Got next segment. Inline: false Dec 29 20:23:28.416310388 (580h54m28.287393423s) [upload=509234]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 20:23:28.416315598 (580h54m28.287398634s) [upload=509234]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 20:23:28.660976111 (580h54m28.532059146s) [seg_pos={0 0}, upload=509234]: Begin upload segment... Dec 29 20:23:28.661026877 (580h54m28.532109922s) [seg_pos={0 0}, upload=509234]: Done begin upload segment. Dec 29 20:23:28.661068466 (580h54m28.532151501s) [noise=true , address=145.79.227.48:4303 , piece_id=KC5NQZIPPNPLQKZL7EYL7RL5Q3HLL4GQQ6S42AHTDENJJ6ZX6ZJA, seg_pos={0 0}, upload=509234]: Uploading piece... Dec 29 20:23:28.661072965 (580h54m28.532156s) [noise=true , address=145.79.227.42:4123 , piece_id=OD5XMSISHKBNLSQNAMV3ZCLBFMRLM7CGS7XKPC2ZU2J5R5AD6GVQ, seg_pos={0 0}, upload=509234]: Uploading piece... Dec 29 20:23:28.661729705 (580h54m28.53281274s) [noise=true , address=145.79.227.39:4014 , piece_id=J2YF6RILHWJI7JLAWCYN4WXBRQMENM63WLRTKNV4H5S3ALX4Q5VQ, seg_pos={0 0}, upload=509234]: Uploading piece... Dec 29 20:23:28.662080493 (580h54m28.533163528s) [noise=true , address=45.91.116.8:4045 , piece_id=46CTNCF4EIA4QSWQRCGTFZRHDOAMFZG5VV3I2WBZYJE7YABX4HEQ, seg_pos={0 0}, upload=509234]: Uploading piece... Dec 29 20:23:28.662340749 (580h54m28.533423784s) [noise=true , address=77.90.31.11:4554 , piece_id=TLQONEXNWGMCRKW2AIZ25JK46DBEYPJGDIFYSQHIYEGHUDYCMG4A, seg_pos={0 0}, upload=509234]: Uploading piece... Dec 29 20:23:28.662600053 (580h54m28.533683098s) [noise=true , address=145.79.227.4:4187 , piece_id=FGSQSSM4EO73VDOYM5AJD2XZSYVU5LWUCOVOG3GRZGRCDUODSLDA, seg_pos={0 0}, upload=509234]: Uploading piece... Dec 29 20:23:28.664093767 (580h54m28.535176802s) [noise=true , address=145.79.227.47:4266 , piece_id=O3QWUUL63R42OY2CBZPPCIPLEMUTE7HJRYHHBOOEOUZU2NQZXXKQ, seg_pos={0 0}, upload=509234]: Uploading piece... Dec 29 20:23:28.664605071 (580h54m28.535688106s) [noise=true , address=145.79.227.35:4481 , piece_id=IUATLJFBJ5HG3H4W44EFZCRJ7U5V3HXJHBQRTMRGYWGAEVQXQX6Q, seg_pos={0 0}, upload=509234]: Uploading piece... Dec 29 20:23:28.665273063 (580h54m28.536356108s) [noise=true , address=77.90.31.21:4878 , piece_id=T6ZKRL26GO7CGZ23I4N56F66YHWUM6XJ6D3AI6VFICGAH6GNICJQ, seg_pos={0 0}, upload=509234]: Uploading piece... Dec 29 20:23:28.665755532 (580h54m28.536838577s) [noise=true , address=45.91.116.6:4974 , piece_id=MY4MH5QULSN3DBEGRFO75W2VHDANQNOHM2F6JNXIQUJXEGCL2J2A, seg_pos={0 0}, upload=509234]: Uploading piece... Dec 29 20:23:28.667333967 (580h54m28.538417012s) [noise=true , address=45.91.116.11:4119 , piece_id=OL6RAER4DLQDSIQFBPQACL36ZAY6YXFCTW3MVELQT44M6MW665LQ, seg_pos={0 0}, upload=509234]: Uploading piece... Dec 29 20:23:28.668065620 (580h54m28.539148665s) [noise=true , address=145.79.227.16:4912 , piece_id=MTCEEL6SYO6DXOXXRNDN243LSYSOFVBLTMMGPFJOBZCJDCEONEPA, seg_pos={0 0}, upload=509234]: Uploading piece... Dec 29 20:23:28.669481887 (580h54m28.540564922s) [noise=true , address=145.79.227.7:4293 , piece_id=LX36RRBBW2YRBQJDRQ4V3FA7TSNI5BVRZW7CCSYJNWDRTCRD5KTA, seg_pos={0 0}, upload=509234]: Uploading piece... Dec 29 20:23:28.670629553 (580h54m28.541712588s) [noise=true , address=145.79.227.41:4085 , piece_id=O26CLJH4ICQNMHBTENAJLRNLDFLHA3CV2RMNGK32IZPWEER6DY2Q, seg_pos={0 0}, upload=509234]: Uploading piece... Dec 29 20:23:28.671866527 (580h54m28.542949562s) [noise=true , address=45.91.116.10:4082 , piece_id=N6HSAF6XCLGUZXDNGBHCGB3CSMQMBNETCFW7CHU7LJGTZ4JJLREA, seg_pos={0 0}, upload=509234]: Uploading piece... Dec 29 20:23:28.672948639 (580h54m28.544031674s) [noise=true , address=45.91.116.12:4226 , piece_id=57WOZLBFS3KLYOTUM34EQ4RHBEOZEZP6BOWUMD2WK2QHDKQS257A, seg_pos={0 0}, upload=509234]: Uploading piece... Dec 29 20:23:30.355599227 (580h54m30.226682262s) [upload=509234]: Next returned nil segment Dec 29 20:23:30.355633262 (580h54m30.226716307s) [upload=509234]: Waiting for error group managing segments... Dec 29 20:23:30.642553302 (580h54m30.513636337s) [noise=true , address=145.79.227.42:4123 , piece_id=OD5XMSISHKBNLSQNAMV3ZCLBFMRLM7CGS7XKPC2ZU2J5R5AD6GVQ, seg_pos={0 0}, upload=509234]: Done uploading piece. err: <nil> Dec 29 20:23:30.647195511 (580h54m30.518278546s) [noise=true , address=145.79.227.4:4187 , piece_id=FGSQSSM4EO73VDOYM5AJD2XZSYVU5LWUCOVOG3GRZGRCDUODSLDA, seg_pos={0 0}, upload=509234]: Done uploading piece. err: <nil> Dec 29 20:23:30.661620258 (580h54m30.532703303s) [noise=true , address=145.79.227.48:4303 , piece_id=KC5NQZIPPNPLQKZL7EYL7RL5Q3HLL4GQQ6S42AHTDENJJ6ZX6ZJA, seg_pos={0 0}, upload=509234]: Done uploading piece. err: <nil> Dec 29 20:23:30.664794151 (580h54m30.535877196s) [noise=true , address=145.79.227.41:4085 , piece_id=O26CLJH4ICQNMHBTENAJLRNLDFLHA3CV2RMNGK32IZPWEER6DY2Q, seg_pos={0 0}, upload=509234]: Done uploading piece. err: <nil> Dec 29 20:23:30.677472743 (580h54m30.548555788s) [noise=true , address=145.79.227.7:4293 , piece_id=LX36RRBBW2YRBQJDRQ4V3FA7TSNI5BVRZW7CCSYJNWDRTCRD5KTA, seg_pos={0 0}, upload=509234]: Done uploading piece. err: <nil> Dec 29 20:23:30.679803722 (580h54m30.550886767s) [noise=true , address=145.79.227.16:4912 , piece_id=MTCEEL6SYO6DXOXXRNDN243LSYSOFVBLTMMGPFJOBZCJDCEONEPA, seg_pos={0 0}, upload=509234]: Done uploading piece. err: <nil> Dec 29 20:23:30.689244619 (580h54m30.560327654s) [noise=true , address=145.79.227.39:4014 , piece_id=J2YF6RILHWJI7JLAWCYN4WXBRQMENM63WLRTKNV4H5S3ALX4Q5VQ, seg_pos={0 0}, upload=509234]: Done uploading piece. err: <nil> Dec 29 20:23:30.700784082 (580h54m30.571867117s) [noise=true , address=145.79.227.47:4266 , piece_id=O3QWUUL63R42OY2CBZPPCIPLEMUTE7HJRYHHBOOEOUZU2NQZXXKQ, seg_pos={0 0}, upload=509234]: Done uploading piece. err: <nil> Dec 29 20:23:30.847297316 (580h54m30.718380351s) [noise=true , address=77.90.31.11:4554 , piece_id=TLQONEXNWGMCRKW2AIZ25JK46DBEYPJGDIFYSQHIYEGHUDYCMG4A, seg_pos={0 0}, upload=509234]: Done uploading piece. err: <nil> Dec 29 20:23:30.920591620 (580h54m30.791674655s) [noise=true , address=45.91.116.10:4082 , piece_id=N6HSAF6XCLGUZXDNGBHCGB3CSMQMBNETCFW7CHU7LJGTZ4JJLREA, seg_pos={0 0}, upload=509234]: Done uploading piece. err: <nil> Dec 29 20:23:30.926928366 (580h54m30.798011401s) [noise=true , address=45.91.116.12:4226 , piece_id=57WOZLBFS3KLYOTUM34EQ4RHBEOZEZP6BOWUMD2WK2QHDKQS257A, seg_pos={0 0}, upload=509234]: Done uploading piece. err: <nil> Dec 29 20:23:30.945981767 (580h54m30.817064812s) [noise=true , address=45.91.116.6:4974 , piece_id=MY4MH5QULSN3DBEGRFO75W2VHDANQNOHM2F6JNXIQUJXEGCL2J2A, seg_pos={0 0}, upload=509234]: Done uploading piece. err: <nil> Dec 29 20:23:30.958572291 (580h54m30.829655326s) [noise=true , address=77.90.31.21:4878 , piece_id=T6ZKRL26GO7CGZ23I4N56F66YHWUM6XJ6D3AI6VFICGAH6GNICJQ, seg_pos={0 0}, upload=509234]: Done uploading piece. err: <nil> Dec 29 20:23:30.958595034 (580h54m30.829678069s) [seg_pos={0 0}, upload=509234]: Segment reached optimal threshold of 13 pieces. Dec 29 20:23:30.959013401 (580h54m30.830096476s) [noise=true , address=145.79.227.35:4481 , piece_id=IUATLJFBJ5HG3H4W44EFZCRJ7U5V3HXJHBQRTMRGYWGAEVQXQX6Q, seg_pos={0 0}, upload=509234]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bRK8WWmAnWUwzCfHu8JtZgrruYSJCvhnsYTKcaLccJcv5RQjp): protocol: expected piece hash; context canceled; EOF Dec 29 20:23:30.959036535 (580h54m30.8301196s) [noise=true , address=145.79.227.35:4481 , piece_id=IUATLJFBJ5HG3H4W44EFZCRJ7U5V3HXJHBQRTMRGYWGAEVQXQX6Q, seg_pos={0 0}, upload=509234]: Long tail cancellation due to optimal threshold Dec 29 20:23:30.959062274 (580h54m30.830145309s) [noise=true , address=45.91.116.11:4119 , piece_id=OL6RAER4DLQDSIQFBPQACL36ZAY6YXFCTW3MVELQT44M6MW665LQ, seg_pos={0 0}, upload=509234]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm): protocol: expected piece hash; context canceled; EOF Dec 29 20:23:30.959070019 (580h54m30.830153054s) [noise=true , address=45.91.116.11:4119 , piece_id=OL6RAER4DLQDSIQFBPQACL36ZAY6YXFCTW3MVELQT44M6MW665LQ, seg_pos={0 0}, upload=509234]: Long tail cancellation due to optimal threshold Dec 29 20:23:30.959092252 (580h54m30.830175297s) [noise=true , address=45.91.116.8:4045 , piece_id=46CTNCF4EIA4QSWQRCGTFZRHDOAMFZG5VV3I2WBZYJE7YABX4HEQ, seg_pos={0 0}, upload=509234]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9): protocol: expected piece hash; context canceled; EOF Dec 29 20:23:30.959097441 (580h54m30.830180476s) [noise=true , address=45.91.116.8:4045 , piece_id=46CTNCF4EIA4QSWQRCGTFZRHDOAMFZG5VV3I2WBZYJE7YABX4HEQ, seg_pos={0 0}, upload=509234]: Long tail cancellation due to optimal threshold Dec 29 20:23:30.959117088 (580h54m30.830200123s) [seg_pos={0 0}, upload=509234]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 20:23:30.959186390 (580h54m30.830269425s) [upload=509234]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 20:23:30.959191369 (580h54m30.830274415s) [upload=509234]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 20:23:31.263824068 (580h54m31.134907103s) [upload=509234]: Done uploading segments... Dec 29 20:23:31.263840989 (580h54m31.134924024s) [upload=509234]: Upload finished. err: <nil> Duration: 6.05674133s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [..X..X.X....X.......X.....................X......X.X.X......] duration: 6.670230057s trace: https://jaeger.dev.storj.io/trace/69a6314bd3cfcdea FAIL (2025-12-29 20:27:54 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, Piece ID: MY4MH5QULSN3DBEGRFO75W2VHDANQNOHM2F6JNXIQUJXEGCL2J2A) context deadline exceeded; piecestore: (Node ID: 12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, Piece ID: 57WOZLBFS3KLYOTUM34EQ4RHBEOZEZP6BOWUMD2WK2QHDKQS257A) context deadline exceeded; piecestore: (Node ID: 1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, Piece ID: TLQONEXNWGMCRKW2AIZ25JK46DBEYPJGDIFYSQHIYEGHUDYCMG4A) context deadline exceeded ======LOGS====== Dec 29 20:23:28.415883495 (580h54m28.28696653s) [upload=509234]: Starting upload Dec 29 20:23:28.415951074 (580h54m28.287034109s) [upload=509234]: Done starting upload Dec 29 20:23:28.416064840 (580h54m28.287147885s) [upload=509234]: Uploading segments... Dec 29 20:23:28.416264120 (580h54m28.287347165s) [upload=509234]: Got next segment. Inline: false Dec 29 20:23:28.416310388 (580h54m28.287393423s) [upload=509234]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 20:23:28.416315598 (580h54m28.287398634s) [upload=509234]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 20:23:28.660976111 (580h54m28.532059146s) [seg_pos={0 0}, upload=509234]: Begin upload segment... Dec 29 20:23:28.661026877 (580h54m28.532109922s) [seg_pos={0 0}, upload=509234]: Done begin upload segment. Dec 29 20:23:28.661068466 (580h54m28.532151501s) [noise=true , address=145.79.227.48:4303 , piece_id=KC5NQZIPPNPLQKZL7EYL7RL5Q3HLL4GQQ6S42AHTDENJJ6ZX6ZJA, seg_pos={0 0}, upload=509234]: Uploading piece... Dec 29 20:23:28.661072965 (580h54m28.532156s) [noise=true , address=145.79.227.42:4123 , piece_id=OD5XMSISHKBNLSQNAMV3ZCLBFMRLM7CGS7XKPC2ZU2J5R5AD6GVQ, seg_pos={0 0}, upload=509234]: Uploading piece... Dec 29 20:23:28.661729705 (580h54m28.53281274s) [noise=true , address=145.79.227.39:4014 , piece_id=J2YF6RILHWJI7JLAWCYN4WXBRQMENM63WLRTKNV4H5S3ALX4Q5VQ, seg_pos={0 0}, upload=509234]: Uploading piece... Dec 29 20:23:28.662080493 (580h54m28.533163528s) [noise=true , address=45.91.116.8:4045 , piece_id=46CTNCF4EIA4QSWQRCGTFZRHDOAMFZG5VV3I2WBZYJE7YABX4HEQ, seg_pos={0 0}, upload=509234]: Uploading piece... Dec 29 20:23:28.662340749 (580h54m28.533423784s) [noise=true , address=77.90.31.11:4554 , piece_id=TLQONEXNWGMCRKW2AIZ25JK46DBEYPJGDIFYSQHIYEGHUDYCMG4A, seg_pos={0 0}, upload=509234]: Uploading piece... Dec 29 20:23:28.662600053 (580h54m28.533683098s) [noise=true , address=145.79.227.4:4187 , piece_id=FGSQSSM4EO73VDOYM5AJD2XZSYVU5LWUCOVOG3GRZGRCDUODSLDA, seg_pos={0 0}, upload=509234]: Uploading piece... Dec 29 20:23:28.664093767 (580h54m28.535176802s) [noise=true , address=145.79.227.47:4266 , piece_id=O3QWUUL63R42OY2CBZPPCIPLEMUTE7HJRYHHBOOEOUZU2NQZXXKQ, seg_pos={0 0}, upload=509234]: Uploading piece... Dec 29 20:23:28.664605071 (580h54m28.535688106s) [noise=true , address=145.79.227.35:4481 , piece_id=IUATLJFBJ5HG3H4W44EFZCRJ7U5V3HXJHBQRTMRGYWGAEVQXQX6Q, seg_pos={0 0}, upload=509234]: Uploading piece... Dec 29 20:23:28.665273063 (580h54m28.536356108s) [noise=true , address=77.90.31.21:4878 , piece_id=T6ZKRL26GO7CGZ23I4N56F66YHWUM6XJ6D3AI6VFICGAH6GNICJQ, seg_pos={0 0}, upload=509234]: Uploading piece... Dec 29 20:23:28.665755532 (580h54m28.536838577s) [noise=true , address=45.91.116.6:4974 , piece_id=MY4MH5QULSN3DBEGRFO75W2VHDANQNOHM2F6JNXIQUJXEGCL2J2A, seg_pos={0 0}, upload=509234]: Uploading piece... Dec 29 20:23:28.667333967 (580h54m28.538417012s) [noise=true , address=45.91.116.11:4119 , piece_id=OL6RAER4DLQDSIQFBPQACL36ZAY6YXFCTW3MVELQT44M6MW665LQ, seg_pos={0 0}, upload=509234]: Uploading piece... Dec 29 20:23:28.668065620 (580h54m28.539148665s) [noise=true , address=145.79.227.16:4912 , piece_id=MTCEEL6SYO6DXOXXRNDN243LSYSOFVBLTMMGPFJOBZCJDCEONEPA, seg_pos={0 0}, upload=509234]: Uploading piece... Dec 29 20:23:28.669481887 (580h54m28.540564922s) [noise=true , address=145.79.227.7:4293 , piece_id=LX36RRBBW2YRBQJDRQ4V3FA7TSNI5BVRZW7CCSYJNWDRTCRD5KTA, seg_pos={0 0}, upload=509234]: Uploading piece... Dec 29 20:23:28.670629553 (580h54m28.541712588s) [noise=true , address=145.79.227.41:4085 , piece_id=O26CLJH4ICQNMHBTENAJLRNLDFLHA3CV2RMNGK32IZPWEER6DY2Q, seg_pos={0 0}, upload=509234]: Uploading piece... Dec 29 20:23:28.671866527 (580h54m28.542949562s) [noise=true , address=45.91.116.10:4082 , piece_id=N6HSAF6XCLGUZXDNGBHCGB3CSMQMBNETCFW7CHU7LJGTZ4JJLREA, seg_pos={0 0}, upload=509234]: Uploading piece... Dec 29 20:23:28.672948639 (580h54m28.544031674s) [noise=true , address=45.91.116.12:4226 , piece_id=57WOZLBFS3KLYOTUM34EQ4RHBEOZEZP6BOWUMD2WK2QHDKQS257A, seg_pos={0 0}, upload=509234]: Uploading piece... Dec 29 20:23:30.355599227 (580h54m30.226682262s) [upload=509234]: Next returned nil segment Dec 29 20:23:30.355633262 (580h54m30.226716307s) [upload=509234]: Waiting for error group managing segments... Dec 29 20:23:30.642553302 (580h54m30.513636337s) [noise=true , address=145.79.227.42:4123 , piece_id=OD5XMSISHKBNLSQNAMV3ZCLBFMRLM7CGS7XKPC2ZU2J5R5AD6GVQ, seg_pos={0 0}, upload=509234]: Done uploading piece. err: <nil> Dec 29 20:23:30.647195511 (580h54m30.518278546s) [noise=true , address=145.79.227.4:4187 , piece_id=FGSQSSM4EO73VDOYM5AJD2XZSYVU5LWUCOVOG3GRZGRCDUODSLDA, seg_pos={0 0}, upload=509234]: Done uploading piece. err: <nil> Dec 29 20:23:30.661620258 (580h54m30.532703303s) [noise=true , address=145.79.227.48:4303 , piece_id=KC5NQZIPPNPLQKZL7EYL7RL5Q3HLL4GQQ6S42AHTDENJJ6ZX6ZJA, seg_pos={0 0}, upload=509234]: Done uploading piece. err: <nil> Dec 29 20:23:30.664794151 (580h54m30.535877196s) [noise=true , address=145.79.227.41:4085 , piece_id=O26CLJH4ICQNMHBTENAJLRNLDFLHA3CV2RMNGK32IZPWEER6DY2Q, seg_pos={0 0}, upload=509234]: Done uploading piece. err: <nil> Dec 29 20:23:30.677472743 (580h54m30.548555788s) [noise=true , address=145.79.227.7:4293 , piece_id=LX36RRBBW2YRBQJDRQ4V3FA7TSNI5BVRZW7CCSYJNWDRTCRD5KTA, seg_pos={0 0}, upload=509234]: Done uploading piece. err: <nil> Dec 29 20:23:30.679803722 (580h54m30.550886767s) [noise=true , address=145.79.227.16:4912 , piece_id=MTCEEL6SYO6DXOXXRNDN243LSYSOFVBLTMMGPFJOBZCJDCEONEPA, seg_pos={0 0}, upload=509234]: Done uploading piece. err: <nil> Dec 29 20:23:30.689244619 (580h54m30.560327654s) [noise=true , address=145.79.227.39:4014 , piece_id=J2YF6RILHWJI7JLAWCYN4WXBRQMENM63WLRTKNV4H5S3ALX4Q5VQ, seg_pos={0 0}, upload=509234]: Done uploading piece. err: <nil> Dec 29 20:23:30.700784082 (580h54m30.571867117s) [noise=true , address=145.79.227.47:4266 , piece_id=O3QWUUL63R42OY2CBZPPCIPLEMUTE7HJRYHHBOOEOUZU2NQZXXKQ, seg_pos={0 0}, upload=509234]: Done uploading piece. err: <nil> Dec 29 20:23:30.847297316 (580h54m30.718380351s) [noise=true , address=77.90.31.11:4554 , piece_id=TLQONEXNWGMCRKW2AIZ25JK46DBEYPJGDIFYSQHIYEGHUDYCMG4A, seg_pos={0 0}, upload=509234]: Done uploading piece. err: <nil> Dec 29 20:23:30.920591620 (580h54m30.791674655s) [noise=true , address=45.91.116.10:4082 , piece_id=N6HSAF6XCLGUZXDNGBHCGB3CSMQMBNETCFW7CHU7LJGTZ4JJLREA, seg_pos={0 0}, upload=509234]: Done uploading piece. err: <nil> Dec 29 20:23:30.926928366 (580h54m30.798011401s) [noise=true , address=45.91.116.12:4226 , piece_id=57WOZLBFS3KLYOTUM34EQ4RHBEOZEZP6BOWUMD2WK2QHDKQS257A, seg_pos={0 0}, upload=509234]: Done uploading piece. err: <nil> Dec 29 20:23:30.945981767 (580h54m30.817064812s) [noise=true , address=45.91.116.6:4974 , piece_id=MY4MH5QULSN3DBEGRFO75W2VHDANQNOHM2F6JNXIQUJXEGCL2J2A, seg_pos={0 0}, upload=509234]: Done uploading piece. err: <nil> Dec 29 20:23:30.958572291 (580h54m30.829655326s) [noise=true , address=77.90.31.21:4878 , piece_id=T6ZKRL26GO7CGZ23I4N56F66YHWUM6XJ6D3AI6VFICGAH6GNICJQ, seg_pos={0 0}, upload=509234]: Done uploading piece. err: <nil> Dec 29 20:23:30.958595034 (580h54m30.829678069s) [seg_pos={0 0}, upload=509234]: Segment reached optimal threshold of 13 pieces. Dec 29 20:23:30.959013401 (580h54m30.830096476s) [noise=true , address=145.79.227.35:4481 , piece_id=IUATLJFBJ5HG3H4W44EFZCRJ7U5V3HXJHBQRTMRGYWGAEVQXQX6Q, seg_pos={0 0}, upload=509234]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bRK8WWmAnWUwzCfHu8JtZgrruYSJCvhnsYTKcaLccJcv5RQjp): protocol: expected piece hash; context canceled; EOF Dec 29 20:23:30.959036535 (580h54m30.8301196s) [noise=true , address=145.79.227.35:4481 , piece_id=IUATLJFBJ5HG3H4W44EFZCRJ7U5V3HXJHBQRTMRGYWGAEVQXQX6Q, seg_pos={0 0}, upload=509234]: Long tail cancellation due to optimal threshold Dec 29 20:23:30.959062274 (580h54m30.830145309s) [noise=true , address=45.91.116.11:4119 , piece_id=OL6RAER4DLQDSIQFBPQACL36ZAY6YXFCTW3MVELQT44M6MW665LQ, seg_pos={0 0}, upload=509234]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm): protocol: expected piece hash; context canceled; EOF Dec 29 20:23:30.959070019 (580h54m30.830153054s) [noise=true , address=45.91.116.11:4119 , piece_id=OL6RAER4DLQDSIQFBPQACL36ZAY6YXFCTW3MVELQT44M6MW665LQ, seg_pos={0 0}, upload=509234]: Long tail cancellation due to optimal threshold Dec 29 20:23:30.959092252 (580h54m30.830175297s) [noise=true , address=45.91.116.8:4045 , piece_id=46CTNCF4EIA4QSWQRCGTFZRHDOAMFZG5VV3I2WBZYJE7YABX4HEQ, seg_pos={0 0}, upload=509234]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9): protocol: expected piece hash; context canceled; EOF Dec 29 20:23:30.959097441 (580h54m30.830180476s) [noise=true , address=45.91.116.8:4045 , piece_id=46CTNCF4EIA4QSWQRCGTFZRHDOAMFZG5VV3I2WBZYJE7YABX4HEQ, seg_pos={0 0}, upload=509234]: Long tail cancellation due to optimal threshold Dec 29 20:23:30.959117088 (580h54m30.830200123s) [seg_pos={0 0}, upload=509234]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 20:23:30.959186390 (580h54m30.830269425s) [upload=509234]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 20:23:30.959191369 (580h54m30.830274415s) [upload=509234]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 20:23:31.263824068 (580h54m31.134907103s) [upload=509234]: Done uploading segments... Dec 29 20:23:31.263840989 (580h54m31.134924024s) [upload=509234]: Upload finished. err: <nil> Duration: 6.670283187s core uplink us-select-1
checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing) antiblip: 1 out of 60 is failed (threshold=6) [................X...........................................] duration: 287.227434ms remote: "tcp:207.211.208.129:443" target: https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt trace: https://jaeger.dev.storj.io/trace/4679501fce62ca7 FAIL (2025-12-29 20:27:39 UTC) orbiter warning: Get "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt": context deadline exceeded Duration: 287.253255ms edge linksharing bgp eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [...X..X.X....X.......X.....................X......X.X.X.....] duration: 17.791740732s trace: https://jaeger.dev.storj.io/trace/69a6314bd3cfcdea FAIL (2025-12-29 20:27:15 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, Piece ID: MY4MH5QULSN3DBEGRFO75W2VHDANQNOHM2F6JNXIQUJXEGCL2J2A) context deadline exceeded; piecestore: (Node ID: 12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, Piece ID: 57WOZLBFS3KLYOTUM34EQ4RHBEOZEZP6BOWUMD2WK2QHDKQS257A) context deadline exceeded; piecestore: (Node ID: 1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, Piece ID: TLQONEXNWGMCRKW2AIZ25JK46DBEYPJGDIFYSQHIYEGHUDYCMG4A) context deadline exceeded ======LOGS====== Dec 29 20:23:28.415883495 (580h54m28.28696653s) [upload=509234]: Starting upload Dec 29 20:23:28.415951074 (580h54m28.287034109s) [upload=509234]: Done starting upload Dec 29 20:23:28.416064840 (580h54m28.287147885s) [upload=509234]: Uploading segments... Dec 29 20:23:28.416264120 (580h54m28.287347165s) [upload=509234]: Got next segment. Inline: false Dec 29 20:23:28.416310388 (580h54m28.287393423s) [upload=509234]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 20:23:28.416315598 (580h54m28.287398634s) [upload=509234]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 20:23:28.660976111 (580h54m28.532059146s) [seg_pos={0 0}, upload=509234]: Begin upload segment... Dec 29 20:23:28.661026877 (580h54m28.532109922s) [seg_pos={0 0}, upload=509234]: Done begin upload segment. Dec 29 20:23:28.661068466 (580h54m28.532151501s) [noise=true , address=145.79.227.48:4303 , piece_id=KC5NQZIPPNPLQKZL7EYL7RL5Q3HLL4GQQ6S42AHTDENJJ6ZX6ZJA, seg_pos={0 0}, upload=509234]: Uploading piece... Dec 29 20:23:28.661072965 (580h54m28.532156s) [noise=true , address=145.79.227.42:4123 , piece_id=OD5XMSISHKBNLSQNAMV3ZCLBFMRLM7CGS7XKPC2ZU2J5R5AD6GVQ, seg_pos={0 0}, upload=509234]: Uploading piece... Dec 29 20:23:28.661729705 (580h54m28.53281274s) [noise=true , address=145.79.227.39:4014 , piece_id=J2YF6RILHWJI7JLAWCYN4WXBRQMENM63WLRTKNV4H5S3ALX4Q5VQ, seg_pos={0 0}, upload=509234]: Uploading piece... Dec 29 20:23:28.662080493 (580h54m28.533163528s) [noise=true , address=45.91.116.8:4045 , piece_id=46CTNCF4EIA4QSWQRCGTFZRHDOAMFZG5VV3I2WBZYJE7YABX4HEQ, seg_pos={0 0}, upload=509234]: Uploading piece... Dec 29 20:23:28.662340749 (580h54m28.533423784s) [noise=true , address=77.90.31.11:4554 , piece_id=TLQONEXNWGMCRKW2AIZ25JK46DBEYPJGDIFYSQHIYEGHUDYCMG4A, seg_pos={0 0}, upload=509234]: Uploading piece... Dec 29 20:23:28.662600053 (580h54m28.533683098s) [noise=true , address=145.79.227.4:4187 , piece_id=FGSQSSM4EO73VDOYM5AJD2XZSYVU5LWUCOVOG3GRZGRCDUODSLDA, seg_pos={0 0}, upload=509234]: Uploading piece... Dec 29 20:23:28.664093767 (580h54m28.535176802s) [noise=true , address=145.79.227.47:4266 , piece_id=O3QWUUL63R42OY2CBZPPCIPLEMUTE7HJRYHHBOOEOUZU2NQZXXKQ, seg_pos={0 0}, upload=509234]: Uploading piece... Dec 29 20:23:28.664605071 (580h54m28.535688106s) [noise=true , address=145.79.227.35:4481 , piece_id=IUATLJFBJ5HG3H4W44EFZCRJ7U5V3HXJHBQRTMRGYWGAEVQXQX6Q, seg_pos={0 0}, upload=509234]: Uploading piece... Dec 29 20:23:28.665273063 (580h54m28.536356108s) [noise=true , address=77.90.31.21:4878 , piece_id=T6ZKRL26GO7CGZ23I4N56F66YHWUM6XJ6D3AI6VFICGAH6GNICJQ, seg_pos={0 0}, upload=509234]: Uploading piece... Dec 29 20:23:28.665755532 (580h54m28.536838577s) [noise=true , address=45.91.116.6:4974 , piece_id=MY4MH5QULSN3DBEGRFO75W2VHDANQNOHM2F6JNXIQUJXEGCL2J2A, seg_pos={0 0}, upload=509234]: Uploading piece... Dec 29 20:23:28.667333967 (580h54m28.538417012s) [noise=true , address=45.91.116.11:4119 , piece_id=OL6RAER4DLQDSIQFBPQACL36ZAY6YXFCTW3MVELQT44M6MW665LQ, seg_pos={0 0}, upload=509234]: Uploading piece... Dec 29 20:23:28.668065620 (580h54m28.539148665s) [noise=true , address=145.79.227.16:4912 , piece_id=MTCEEL6SYO6DXOXXRNDN243LSYSOFVBLTMMGPFJOBZCJDCEONEPA, seg_pos={0 0}, upload=509234]: Uploading piece... Dec 29 20:23:28.669481887 (580h54m28.540564922s) [noise=true , address=145.79.227.7:4293 , piece_id=LX36RRBBW2YRBQJDRQ4V3FA7TSNI5BVRZW7CCSYJNWDRTCRD5KTA, seg_pos={0 0}, upload=509234]: Uploading piece... Dec 29 20:23:28.670629553 (580h54m28.541712588s) [noise=true , address=145.79.227.41:4085 , piece_id=O26CLJH4ICQNMHBTENAJLRNLDFLHA3CV2RMNGK32IZPWEER6DY2Q, seg_pos={0 0}, upload=509234]: Uploading piece... Dec 29 20:23:28.671866527 (580h54m28.542949562s) [noise=true , address=45.91.116.10:4082 , piece_id=N6HSAF6XCLGUZXDNGBHCGB3CSMQMBNETCFW7CHU7LJGTZ4JJLREA, seg_pos={0 0}, upload=509234]: Uploading piece... Dec 29 20:23:28.672948639 (580h54m28.544031674s) [noise=true , address=45.91.116.12:4226 , piece_id=57WOZLBFS3KLYOTUM34EQ4RHBEOZEZP6BOWUMD2WK2QHDKQS257A, seg_pos={0 0}, upload=509234]: Uploading piece... Dec 29 20:23:30.355599227 (580h54m30.226682262s) [upload=509234]: Next returned nil segment Dec 29 20:23:30.355633262 (580h54m30.226716307s) [upload=509234]: Waiting for error group managing segments... Dec 29 20:23:30.642553302 (580h54m30.513636337s) [noise=true , address=145.79.227.42:4123 , piece_id=OD5XMSISHKBNLSQNAMV3ZCLBFMRLM7CGS7XKPC2ZU2J5R5AD6GVQ, seg_pos={0 0}, upload=509234]: Done uploading piece. err: <nil> Dec 29 20:23:30.647195511 (580h54m30.518278546s) [noise=true , address=145.79.227.4:4187 , piece_id=FGSQSSM4EO73VDOYM5AJD2XZSYVU5LWUCOVOG3GRZGRCDUODSLDA, seg_pos={0 0}, upload=509234]: Done uploading piece. err: <nil> Dec 29 20:23:30.661620258 (580h54m30.532703303s) [noise=true , address=145.79.227.48:4303 , piece_id=KC5NQZIPPNPLQKZL7EYL7RL5Q3HLL4GQQ6S42AHTDENJJ6ZX6ZJA, seg_pos={0 0}, upload=509234]: Done uploading piece. err: <nil> Dec 29 20:23:30.664794151 (580h54m30.535877196s) [noise=true , address=145.79.227.41:4085 , piece_id=O26CLJH4ICQNMHBTENAJLRNLDFLHA3CV2RMNGK32IZPWEER6DY2Q, seg_pos={0 0}, upload=509234]: Done uploading piece. err: <nil> Dec 29 20:23:30.677472743 (580h54m30.548555788s) [noise=true , address=145.79.227.7:4293 , piece_id=LX36RRBBW2YRBQJDRQ4V3FA7TSNI5BVRZW7CCSYJNWDRTCRD5KTA, seg_pos={0 0}, upload=509234]: Done uploading piece. err: <nil> Dec 29 20:23:30.679803722 (580h54m30.550886767s) [noise=true , address=145.79.227.16:4912 , piece_id=MTCEEL6SYO6DXOXXRNDN243LSYSOFVBLTMMGPFJOBZCJDCEONEPA, seg_pos={0 0}, upload=509234]: Done uploading piece. err: <nil> Dec 29 20:23:30.689244619 (580h54m30.560327654s) [noise=true , address=145.79.227.39:4014 , piece_id=J2YF6RILHWJI7JLAWCYN4WXBRQMENM63WLRTKNV4H5S3ALX4Q5VQ, seg_pos={0 0}, upload=509234]: Done uploading piece. err: <nil> Dec 29 20:23:30.700784082 (580h54m30.571867117s) [noise=true , address=145.79.227.47:4266 , piece_id=O3QWUUL63R42OY2CBZPPCIPLEMUTE7HJRYHHBOOEOUZU2NQZXXKQ, seg_pos={0 0}, upload=509234]: Done uploading piece. err: <nil> Dec 29 20:23:30.847297316 (580h54m30.718380351s) [noise=true , address=77.90.31.11:4554 , piece_id=TLQONEXNWGMCRKW2AIZ25JK46DBEYPJGDIFYSQHIYEGHUDYCMG4A, seg_pos={0 0}, upload=509234]: Done uploading piece. err: <nil> Dec 29 20:23:30.920591620 (580h54m30.791674655s) [noise=true , address=45.91.116.10:4082 , piece_id=N6HSAF6XCLGUZXDNGBHCGB3CSMQMBNETCFW7CHU7LJGTZ4JJLREA, seg_pos={0 0}, upload=509234]: Done uploading piece. err: <nil> Dec 29 20:23:30.926928366 (580h54m30.798011401s) [noise=true , address=45.91.116.12:4226 , piece_id=57WOZLBFS3KLYOTUM34EQ4RHBEOZEZP6BOWUMD2WK2QHDKQS257A, seg_pos={0 0}, upload=509234]: Done uploading piece. err: <nil> Dec 29 20:23:30.945981767 (580h54m30.817064812s) [noise=true , address=45.91.116.6:4974 , piece_id=MY4MH5QULSN3DBEGRFO75W2VHDANQNOHM2F6JNXIQUJXEGCL2J2A, seg_pos={0 0}, upload=509234]: Done uploading piece. err: <nil> Dec 29 20:23:30.958572291 (580h54m30.829655326s) [noise=true , address=77.90.31.21:4878 , piece_id=T6ZKRL26GO7CGZ23I4N56F66YHWUM6XJ6D3AI6VFICGAH6GNICJQ, seg_pos={0 0}, upload=509234]: Done uploading piece. err: <nil> Dec 29 20:23:30.958595034 (580h54m30.829678069s) [seg_pos={0 0}, upload=509234]: Segment reached optimal threshold of 13 pieces. Dec 29 20:23:30.959013401 (580h54m30.830096476s) [noise=true , address=145.79.227.35:4481 , piece_id=IUATLJFBJ5HG3H4W44EFZCRJ7U5V3HXJHBQRTMRGYWGAEVQXQX6Q, seg_pos={0 0}, upload=509234]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bRK8WWmAnWUwzCfHu8JtZgrruYSJCvhnsYTKcaLccJcv5RQjp): protocol: expected piece hash; context canceled; EOF Dec 29 20:23:30.959036535 (580h54m30.8301196s) [noise=true , address=145.79.227.35:4481 , piece_id=IUATLJFBJ5HG3H4W44EFZCRJ7U5V3HXJHBQRTMRGYWGAEVQXQX6Q, seg_pos={0 0}, upload=509234]: Long tail cancellation due to optimal threshold Dec 29 20:23:30.959062274 (580h54m30.830145309s) [noise=true , address=45.91.116.11:4119 , piece_id=OL6RAER4DLQDSIQFBPQACL36ZAY6YXFCTW3MVELQT44M6MW665LQ, seg_pos={0 0}, upload=509234]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm): protocol: expected piece hash; context canceled; EOF Dec 29 20:23:30.959070019 (580h54m30.830153054s) [noise=true , address=45.91.116.11:4119 , piece_id=OL6RAER4DLQDSIQFBPQACL36ZAY6YXFCTW3MVELQT44M6MW665LQ, seg_pos={0 0}, upload=509234]: Long tail cancellation due to optimal threshold Dec 29 20:23:30.959092252 (580h54m30.830175297s) [noise=true , address=45.91.116.8:4045 , piece_id=46CTNCF4EIA4QSWQRCGTFZRHDOAMFZG5VV3I2WBZYJE7YABX4HEQ, seg_pos={0 0}, upload=509234]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9): protocol: expected piece hash; context canceled; EOF Dec 29 20:23:30.959097441 (580h54m30.830180476s) [noise=true , address=45.91.116.8:4045 , piece_id=46CTNCF4EIA4QSWQRCGTFZRHDOAMFZG5VV3I2WBZYJE7YABX4HEQ, seg_pos={0 0}, upload=509234]: Long tail cancellation due to optimal threshold Dec 29 20:23:30.959117088 (580h54m30.830200123s) [seg_pos={0 0}, upload=509234]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 20:23:30.959186390 (580h54m30.830269425s) [upload=509234]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 20:23:30.959191369 (580h54m30.830274415s) [upload=509234]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 20:23:31.263824068 (580h54m31.134907103s) [upload=509234]: Done uploading segments... Dec 29 20:23:31.263840989 (580h54m31.134924024s) [upload=509234]: Upload finished. err: <nil> Duration: 17.791771479s core uplink us-select-1
checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing) antiblip: 1 out of 60 is failed (threshold=6) [.................X..........................................] duration: 250.36281ms remote: "tcp:207.211.208.129:443" target: https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt trace: https://jaeger.dev.storj.io/trace/4679501fce62ca7 FAIL (2025-12-29 20:26:52 UTC) orbiter warning: Get "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt": context deadline exceeded Duration: 250.387566ms edge linksharing bgp eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [....X..X.X....X.......X.....................X......X.X.X....] duration: 5.995691486s trace: https://jaeger.dev.storj.io/trace/69a6314bd3cfcdea FAIL (2025-12-29 20:26:28 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, Piece ID: MY4MH5QULSN3DBEGRFO75W2VHDANQNOHM2F6JNXIQUJXEGCL2J2A) context deadline exceeded; piecestore: (Node ID: 12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, Piece ID: 57WOZLBFS3KLYOTUM34EQ4RHBEOZEZP6BOWUMD2WK2QHDKQS257A) context deadline exceeded; piecestore: (Node ID: 1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, Piece ID: TLQONEXNWGMCRKW2AIZ25JK46DBEYPJGDIFYSQHIYEGHUDYCMG4A) context deadline exceeded ======LOGS====== Dec 29 20:23:28.415883495 (580h54m28.28696653s) [upload=509234]: Starting upload Dec 29 20:23:28.415951074 (580h54m28.287034109s) [upload=509234]: Done starting upload Dec 29 20:23:28.416064840 (580h54m28.287147885s) [upload=509234]: Uploading segments... Dec 29 20:23:28.416264120 (580h54m28.287347165s) [upload=509234]: Got next segment. Inline: false Dec 29 20:23:28.416310388 (580h54m28.287393423s) [upload=509234]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 20:23:28.416315598 (580h54m28.287398634s) [upload=509234]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 20:23:28.660976111 (580h54m28.532059146s) [seg_pos={0 0}, upload=509234]: Begin upload segment... Dec 29 20:23:28.661026877 (580h54m28.532109922s) [seg_pos={0 0}, upload=509234]: Done begin upload segment. Dec 29 20:23:28.661068466 (580h54m28.532151501s) [noise=true , address=145.79.227.48:4303 , piece_id=KC5NQZIPPNPLQKZL7EYL7RL5Q3HLL4GQQ6S42AHTDENJJ6ZX6ZJA, seg_pos={0 0}, upload=509234]: Uploading piece... Dec 29 20:23:28.661072965 (580h54m28.532156s) [noise=true , address=145.79.227.42:4123 , piece_id=OD5XMSISHKBNLSQNAMV3ZCLBFMRLM7CGS7XKPC2ZU2J5R5AD6GVQ, seg_pos={0 0}, upload=509234]: Uploading piece... Dec 29 20:23:28.661729705 (580h54m28.53281274s) [noise=true , address=145.79.227.39:4014 , piece_id=J2YF6RILHWJI7JLAWCYN4WXBRQMENM63WLRTKNV4H5S3ALX4Q5VQ, seg_pos={0 0}, upload=509234]: Uploading piece... Dec 29 20:23:28.662080493 (580h54m28.533163528s) [noise=true , address=45.91.116.8:4045 , piece_id=46CTNCF4EIA4QSWQRCGTFZRHDOAMFZG5VV3I2WBZYJE7YABX4HEQ, seg_pos={0 0}, upload=509234]: Uploading piece... Dec 29 20:23:28.662340749 (580h54m28.533423784s) [noise=true , address=77.90.31.11:4554 , piece_id=TLQONEXNWGMCRKW2AIZ25JK46DBEYPJGDIFYSQHIYEGHUDYCMG4A, seg_pos={0 0}, upload=509234]: Uploading piece... Dec 29 20:23:28.662600053 (580h54m28.533683098s) [noise=true , address=145.79.227.4:4187 , piece_id=FGSQSSM4EO73VDOYM5AJD2XZSYVU5LWUCOVOG3GRZGRCDUODSLDA, seg_pos={0 0}, upload=509234]: Uploading piece... Dec 29 20:23:28.664093767 (580h54m28.535176802s) [noise=true , address=145.79.227.47:4266 , piece_id=O3QWUUL63R42OY2CBZPPCIPLEMUTE7HJRYHHBOOEOUZU2NQZXXKQ, seg_pos={0 0}, upload=509234]: Uploading piece... Dec 29 20:23:28.664605071 (580h54m28.535688106s) [noise=true , address=145.79.227.35:4481 , piece_id=IUATLJFBJ5HG3H4W44EFZCRJ7U5V3HXJHBQRTMRGYWGAEVQXQX6Q, seg_pos={0 0}, upload=509234]: Uploading piece... Dec 29 20:23:28.665273063 (580h54m28.536356108s) [noise=true , address=77.90.31.21:4878 , piece_id=T6ZKRL26GO7CGZ23I4N56F66YHWUM6XJ6D3AI6VFICGAH6GNICJQ, seg_pos={0 0}, upload=509234]: Uploading piece... Dec 29 20:23:28.665755532 (580h54m28.536838577s) [noise=true , address=45.91.116.6:4974 , piece_id=MY4MH5QULSN3DBEGRFO75W2VHDANQNOHM2F6JNXIQUJXEGCL2J2A, seg_pos={0 0}, upload=509234]: Uploading piece... Dec 29 20:23:28.667333967 (580h54m28.538417012s) [noise=true , address=45.91.116.11:4119 , piece_id=OL6RAER4DLQDSIQFBPQACL36ZAY6YXFCTW3MVELQT44M6MW665LQ, seg_pos={0 0}, upload=509234]: Uploading piece... Dec 29 20:23:28.668065620 (580h54m28.539148665s) [noise=true , address=145.79.227.16:4912 , piece_id=MTCEEL6SYO6DXOXXRNDN243LSYSOFVBLTMMGPFJOBZCJDCEONEPA, seg_pos={0 0}, upload=509234]: Uploading piece... Dec 29 20:23:28.669481887 (580h54m28.540564922s) [noise=true , address=145.79.227.7:4293 , piece_id=LX36RRBBW2YRBQJDRQ4V3FA7TSNI5BVRZW7CCSYJNWDRTCRD5KTA, seg_pos={0 0}, upload=509234]: Uploading piece... Dec 29 20:23:28.670629553 (580h54m28.541712588s) [noise=true , address=145.79.227.41:4085 , piece_id=O26CLJH4ICQNMHBTENAJLRNLDFLHA3CV2RMNGK32IZPWEER6DY2Q, seg_pos={0 0}, upload=509234]: Uploading piece... Dec 29 20:23:28.671866527 (580h54m28.542949562s) [noise=true , address=45.91.116.10:4082 , piece_id=N6HSAF6XCLGUZXDNGBHCGB3CSMQMBNETCFW7CHU7LJGTZ4JJLREA, seg_pos={0 0}, upload=509234]: Uploading piece... Dec 29 20:23:28.672948639 (580h54m28.544031674s) [noise=true , address=45.91.116.12:4226 , piece_id=57WOZLBFS3KLYOTUM34EQ4RHBEOZEZP6BOWUMD2WK2QHDKQS257A, seg_pos={0 0}, upload=509234]: Uploading piece... Dec 29 20:23:30.355599227 (580h54m30.226682262s) [upload=509234]: Next returned nil segment Dec 29 20:23:30.355633262 (580h54m30.226716307s) [upload=509234]: Waiting for error group managing segments... Dec 29 20:23:30.642553302 (580h54m30.513636337s) [noise=true , address=145.79.227.42:4123 , piece_id=OD5XMSISHKBNLSQNAMV3ZCLBFMRLM7CGS7XKPC2ZU2J5R5AD6GVQ, seg_pos={0 0}, upload=509234]: Done uploading piece. err: <nil> Dec 29 20:23:30.647195511 (580h54m30.518278546s) [noise=true , address=145.79.227.4:4187 , piece_id=FGSQSSM4EO73VDOYM5AJD2XZSYVU5LWUCOVOG3GRZGRCDUODSLDA, seg_pos={0 0}, upload=509234]: Done uploading piece. err: <nil> Dec 29 20:23:30.661620258 (580h54m30.532703303s) [noise=true , address=145.79.227.48:4303 , piece_id=KC5NQZIPPNPLQKZL7EYL7RL5Q3HLL4GQQ6S42AHTDENJJ6ZX6ZJA, seg_pos={0 0}, upload=509234]: Done uploading piece. err: <nil> Dec 29 20:23:30.664794151 (580h54m30.535877196s) [noise=true , address=145.79.227.41:4085 , piece_id=O26CLJH4ICQNMHBTENAJLRNLDFLHA3CV2RMNGK32IZPWEER6DY2Q, seg_pos={0 0}, upload=509234]: Done uploading piece. err: <nil> Dec 29 20:23:30.677472743 (580h54m30.548555788s) [noise=true , address=145.79.227.7:4293 , piece_id=LX36RRBBW2YRBQJDRQ4V3FA7TSNI5BVRZW7CCSYJNWDRTCRD5KTA, seg_pos={0 0}, upload=509234]: Done uploading piece. err: <nil> Dec 29 20:23:30.679803722 (580h54m30.550886767s) [noise=true , address=145.79.227.16:4912 , piece_id=MTCEEL6SYO6DXOXXRNDN243LSYSOFVBLTMMGPFJOBZCJDCEONEPA, seg_pos={0 0}, upload=509234]: Done uploading piece. err: <nil> Dec 29 20:23:30.689244619 (580h54m30.560327654s) [noise=true , address=145.79.227.39:4014 , piece_id=J2YF6RILHWJI7JLAWCYN4WXBRQMENM63WLRTKNV4H5S3ALX4Q5VQ, seg_pos={0 0}, upload=509234]: Done uploading piece. err: <nil> Dec 29 20:23:30.700784082 (580h54m30.571867117s) [noise=true , address=145.79.227.47:4266 , piece_id=O3QWUUL63R42OY2CBZPPCIPLEMUTE7HJRYHHBOOEOUZU2NQZXXKQ, seg_pos={0 0}, upload=509234]: Done uploading piece. err: <nil> Dec 29 20:23:30.847297316 (580h54m30.718380351s) [noise=true , address=77.90.31.11:4554 , piece_id=TLQONEXNWGMCRKW2AIZ25JK46DBEYPJGDIFYSQHIYEGHUDYCMG4A, seg_pos={0 0}, upload=509234]: Done uploading piece. err: <nil> Dec 29 20:23:30.920591620 (580h54m30.791674655s) [noise=true , address=45.91.116.10:4082 , piece_id=N6HSAF6XCLGUZXDNGBHCGB3CSMQMBNETCFW7CHU7LJGTZ4JJLREA, seg_pos={0 0}, upload=509234]: Done uploading piece. err: <nil> Dec 29 20:23:30.926928366 (580h54m30.798011401s) [noise=true , address=45.91.116.12:4226 , piece_id=57WOZLBFS3KLYOTUM34EQ4RHBEOZEZP6BOWUMD2WK2QHDKQS257A, seg_pos={0 0}, upload=509234]: Done uploading piece. err: <nil> Dec 29 20:23:30.945981767 (580h54m30.817064812s) [noise=true , address=45.91.116.6:4974 , piece_id=MY4MH5QULSN3DBEGRFO75W2VHDANQNOHM2F6JNXIQUJXEGCL2J2A, seg_pos={0 0}, upload=509234]: Done uploading piece. err: <nil> Dec 29 20:23:30.958572291 (580h54m30.829655326s) [noise=true , address=77.90.31.21:4878 , piece_id=T6ZKRL26GO7CGZ23I4N56F66YHWUM6XJ6D3AI6VFICGAH6GNICJQ, seg_pos={0 0}, upload=509234]: Done uploading piece. err: <nil> Dec 29 20:23:30.958595034 (580h54m30.829678069s) [seg_pos={0 0}, upload=509234]: Segment reached optimal threshold of 13 pieces. Dec 29 20:23:30.959013401 (580h54m30.830096476s) [noise=true , address=145.79.227.35:4481 , piece_id=IUATLJFBJ5HG3H4W44EFZCRJ7U5V3HXJHBQRTMRGYWGAEVQXQX6Q, seg_pos={0 0}, upload=509234]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bRK8WWmAnWUwzCfHu8JtZgrruYSJCvhnsYTKcaLccJcv5RQjp): protocol: expected piece hash; context canceled; EOF Dec 29 20:23:30.959036535 (580h54m30.8301196s) [noise=true , address=145.79.227.35:4481 , piece_id=IUATLJFBJ5HG3H4W44EFZCRJ7U5V3HXJHBQRTMRGYWGAEVQXQX6Q, seg_pos={0 0}, upload=509234]: Long tail cancellation due to optimal threshold Dec 29 20:23:30.959062274 (580h54m30.830145309s) [noise=true , address=45.91.116.11:4119 , piece_id=OL6RAER4DLQDSIQFBPQACL36ZAY6YXFCTW3MVELQT44M6MW665LQ, seg_pos={0 0}, upload=509234]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm): protocol: expected piece hash; context canceled; EOF Dec 29 20:23:30.959070019 (580h54m30.830153054s) [noise=true , address=45.91.116.11:4119 , piece_id=OL6RAER4DLQDSIQFBPQACL36ZAY6YXFCTW3MVELQT44M6MW665LQ, seg_pos={0 0}, upload=509234]: Long tail cancellation due to optimal threshold Dec 29 20:23:30.959092252 (580h54m30.830175297s) [noise=true , address=45.91.116.8:4045 , piece_id=46CTNCF4EIA4QSWQRCGTFZRHDOAMFZG5VV3I2WBZYJE7YABX4HEQ, seg_pos={0 0}, upload=509234]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9): protocol: expected piece hash; context canceled; EOF Dec 29 20:23:30.959097441 (580h54m30.830180476s) [noise=true , address=45.91.116.8:4045 , piece_id=46CTNCF4EIA4QSWQRCGTFZRHDOAMFZG5VV3I2WBZYJE7YABX4HEQ, seg_pos={0 0}, upload=509234]: Long tail cancellation due to optimal threshold Dec 29 20:23:30.959117088 (580h54m30.830200123s) [seg_pos={0 0}, upload=509234]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 20:23:30.959186390 (580h54m30.830269425s) [upload=509234]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 20:23:30.959191369 (580h54m30.830274415s) [upload=509234]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 20:23:31.263824068 (580h54m31.134907103s) [upload=509234]: Done uploading segments... Dec 29 20:23:31.263840989 (580h54m31.134924024s) [upload=509234]: Upload finished. err: <nil> Duration: 5.995718967s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [X....X..X.X....X.......X.....................X......X.X.X...] duration: 6.015312005s trace: https://jaeger.dev.storj.io/trace/69a6314bd3cfcdea FAIL (2025-12-29 20:25:51 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, Piece ID: MY4MH5QULSN3DBEGRFO75W2VHDANQNOHM2F6JNXIQUJXEGCL2J2A) context deadline exceeded; piecestore: (Node ID: 12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, Piece ID: 57WOZLBFS3KLYOTUM34EQ4RHBEOZEZP6BOWUMD2WK2QHDKQS257A) context deadline exceeded; piecestore: (Node ID: 1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, Piece ID: TLQONEXNWGMCRKW2AIZ25JK46DBEYPJGDIFYSQHIYEGHUDYCMG4A) context deadline exceeded ======LOGS====== Dec 29 20:23:28.415883495 (580h54m28.28696653s) [upload=509234]: Starting upload Dec 29 20:23:28.415951074 (580h54m28.287034109s) [upload=509234]: Done starting upload Dec 29 20:23:28.416064840 (580h54m28.287147885s) [upload=509234]: Uploading segments... Dec 29 20:23:28.416264120 (580h54m28.287347165s) [upload=509234]: Got next segment. Inline: false Dec 29 20:23:28.416310388 (580h54m28.287393423s) [upload=509234]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 20:23:28.416315598 (580h54m28.287398634s) [upload=509234]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 20:23:28.660976111 (580h54m28.532059146s) [seg_pos={0 0}, upload=509234]: Begin upload segment... Dec 29 20:23:28.661026877 (580h54m28.532109922s) [seg_pos={0 0}, upload=509234]: Done begin upload segment. Dec 29 20:23:28.661068466 (580h54m28.532151501s) [noise=true , address=145.79.227.48:4303 , piece_id=KC5NQZIPPNPLQKZL7EYL7RL5Q3HLL4GQQ6S42AHTDENJJ6ZX6ZJA, seg_pos={0 0}, upload=509234]: Uploading piece... Dec 29 20:23:28.661072965 (580h54m28.532156s) [noise=true , address=145.79.227.42:4123 , piece_id=OD5XMSISHKBNLSQNAMV3ZCLBFMRLM7CGS7XKPC2ZU2J5R5AD6GVQ, seg_pos={0 0}, upload=509234]: Uploading piece... Dec 29 20:23:28.661729705 (580h54m28.53281274s) [noise=true , address=145.79.227.39:4014 , piece_id=J2YF6RILHWJI7JLAWCYN4WXBRQMENM63WLRTKNV4H5S3ALX4Q5VQ, seg_pos={0 0}, upload=509234]: Uploading piece... Dec 29 20:23:28.662080493 (580h54m28.533163528s) [noise=true , address=45.91.116.8:4045 , piece_id=46CTNCF4EIA4QSWQRCGTFZRHDOAMFZG5VV3I2WBZYJE7YABX4HEQ, seg_pos={0 0}, upload=509234]: Uploading piece... Dec 29 20:23:28.662340749 (580h54m28.533423784s) [noise=true , address=77.90.31.11:4554 , piece_id=TLQONEXNWGMCRKW2AIZ25JK46DBEYPJGDIFYSQHIYEGHUDYCMG4A, seg_pos={0 0}, upload=509234]: Uploading piece... Dec 29 20:23:28.662600053 (580h54m28.533683098s) [noise=true , address=145.79.227.4:4187 , piece_id=FGSQSSM4EO73VDOYM5AJD2XZSYVU5LWUCOVOG3GRZGRCDUODSLDA, seg_pos={0 0}, upload=509234]: Uploading piece... Dec 29 20:23:28.664093767 (580h54m28.535176802s) [noise=true , address=145.79.227.47:4266 , piece_id=O3QWUUL63R42OY2CBZPPCIPLEMUTE7HJRYHHBOOEOUZU2NQZXXKQ, seg_pos={0 0}, upload=509234]: Uploading piece... Dec 29 20:23:28.664605071 (580h54m28.535688106s) [noise=true , address=145.79.227.35:4481 , piece_id=IUATLJFBJ5HG3H4W44EFZCRJ7U5V3HXJHBQRTMRGYWGAEVQXQX6Q, seg_pos={0 0}, upload=509234]: Uploading piece... Dec 29 20:23:28.665273063 (580h54m28.536356108s) [noise=true , address=77.90.31.21:4878 , piece_id=T6ZKRL26GO7CGZ23I4N56F66YHWUM6XJ6D3AI6VFICGAH6GNICJQ, seg_pos={0 0}, upload=509234]: Uploading piece... Dec 29 20:23:28.665755532 (580h54m28.536838577s) [noise=true , address=45.91.116.6:4974 , piece_id=MY4MH5QULSN3DBEGRFO75W2VHDANQNOHM2F6JNXIQUJXEGCL2J2A, seg_pos={0 0}, upload=509234]: Uploading piece... Dec 29 20:23:28.667333967 (580h54m28.538417012s) [noise=true , address=45.91.116.11:4119 , piece_id=OL6RAER4DLQDSIQFBPQACL36ZAY6YXFCTW3MVELQT44M6MW665LQ, seg_pos={0 0}, upload=509234]: Uploading piece... Dec 29 20:23:28.668065620 (580h54m28.539148665s) [noise=true , address=145.79.227.16:4912 , piece_id=MTCEEL6SYO6DXOXXRNDN243LSYSOFVBLTMMGPFJOBZCJDCEONEPA, seg_pos={0 0}, upload=509234]: Uploading piece... Dec 29 20:23:28.669481887 (580h54m28.540564922s) [noise=true , address=145.79.227.7:4293 , piece_id=LX36RRBBW2YRBQJDRQ4V3FA7TSNI5BVRZW7CCSYJNWDRTCRD5KTA, seg_pos={0 0}, upload=509234]: Uploading piece... Dec 29 20:23:28.670629553 (580h54m28.541712588s) [noise=true , address=145.79.227.41:4085 , piece_id=O26CLJH4ICQNMHBTENAJLRNLDFLHA3CV2RMNGK32IZPWEER6DY2Q, seg_pos={0 0}, upload=509234]: Uploading piece... Dec 29 20:23:28.671866527 (580h54m28.542949562s) [noise=true , address=45.91.116.10:4082 , piece_id=N6HSAF6XCLGUZXDNGBHCGB3CSMQMBNETCFW7CHU7LJGTZ4JJLREA, seg_pos={0 0}, upload=509234]: Uploading piece... Dec 29 20:23:28.672948639 (580h54m28.544031674s) [noise=true , address=45.91.116.12:4226 , piece_id=57WOZLBFS3KLYOTUM34EQ4RHBEOZEZP6BOWUMD2WK2QHDKQS257A, seg_pos={0 0}, upload=509234]: Uploading piece... Dec 29 20:23:30.355599227 (580h54m30.226682262s) [upload=509234]: Next returned nil segment Dec 29 20:23:30.355633262 (580h54m30.226716307s) [upload=509234]: Waiting for error group managing segments... Dec 29 20:23:30.642553302 (580h54m30.513636337s) [noise=true , address=145.79.227.42:4123 , piece_id=OD5XMSISHKBNLSQNAMV3ZCLBFMRLM7CGS7XKPC2ZU2J5R5AD6GVQ, seg_pos={0 0}, upload=509234]: Done uploading piece. err: <nil> Dec 29 20:23:30.647195511 (580h54m30.518278546s) [noise=true , address=145.79.227.4:4187 , piece_id=FGSQSSM4EO73VDOYM5AJD2XZSYVU5LWUCOVOG3GRZGRCDUODSLDA, seg_pos={0 0}, upload=509234]: Done uploading piece. err: <nil> Dec 29 20:23:30.661620258 (580h54m30.532703303s) [noise=true , address=145.79.227.48:4303 , piece_id=KC5NQZIPPNPLQKZL7EYL7RL5Q3HLL4GQQ6S42AHTDENJJ6ZX6ZJA, seg_pos={0 0}, upload=509234]: Done uploading piece. err: <nil> Dec 29 20:23:30.664794151 (580h54m30.535877196s) [noise=true , address=145.79.227.41:4085 , piece_id=O26CLJH4ICQNMHBTENAJLRNLDFLHA3CV2RMNGK32IZPWEER6DY2Q, seg_pos={0 0}, upload=509234]: Done uploading piece. err: <nil> Dec 29 20:23:30.677472743 (580h54m30.548555788s) [noise=true , address=145.79.227.7:4293 , piece_id=LX36RRBBW2YRBQJDRQ4V3FA7TSNI5BVRZW7CCSYJNWDRTCRD5KTA, seg_pos={0 0}, upload=509234]: Done uploading piece. err: <nil> Dec 29 20:23:30.679803722 (580h54m30.550886767s) [noise=true , address=145.79.227.16:4912 , piece_id=MTCEEL6SYO6DXOXXRNDN243LSYSOFVBLTMMGPFJOBZCJDCEONEPA, seg_pos={0 0}, upload=509234]: Done uploading piece. err: <nil> Dec 29 20:23:30.689244619 (580h54m30.560327654s) [noise=true , address=145.79.227.39:4014 , piece_id=J2YF6RILHWJI7JLAWCYN4WXBRQMENM63WLRTKNV4H5S3ALX4Q5VQ, seg_pos={0 0}, upload=509234]: Done uploading piece. err: <nil> Dec 29 20:23:30.700784082 (580h54m30.571867117s) [noise=true , address=145.79.227.47:4266 , piece_id=O3QWUUL63R42OY2CBZPPCIPLEMUTE7HJRYHHBOOEOUZU2NQZXXKQ, seg_pos={0 0}, upload=509234]: Done uploading piece. err: <nil> Dec 29 20:23:30.847297316 (580h54m30.718380351s) [noise=true , address=77.90.31.11:4554 , piece_id=TLQONEXNWGMCRKW2AIZ25JK46DBEYPJGDIFYSQHIYEGHUDYCMG4A, seg_pos={0 0}, upload=509234]: Done uploading piece. err: <nil> Dec 29 20:23:30.920591620 (580h54m30.791674655s) [noise=true , address=45.91.116.10:4082 , piece_id=N6HSAF6XCLGUZXDNGBHCGB3CSMQMBNETCFW7CHU7LJGTZ4JJLREA, seg_pos={0 0}, upload=509234]: Done uploading piece. err: <nil> Dec 29 20:23:30.926928366 (580h54m30.798011401s) [noise=true , address=45.91.116.12:4226 , piece_id=57WOZLBFS3KLYOTUM34EQ4RHBEOZEZP6BOWUMD2WK2QHDKQS257A, seg_pos={0 0}, upload=509234]: Done uploading piece. err: <nil> Dec 29 20:23:30.945981767 (580h54m30.817064812s) [noise=true , address=45.91.116.6:4974 , piece_id=MY4MH5QULSN3DBEGRFO75W2VHDANQNOHM2F6JNXIQUJXEGCL2J2A, seg_pos={0 0}, upload=509234]: Done uploading piece. err: <nil> Dec 29 20:23:30.958572291 (580h54m30.829655326s) [noise=true , address=77.90.31.21:4878 , piece_id=T6ZKRL26GO7CGZ23I4N56F66YHWUM6XJ6D3AI6VFICGAH6GNICJQ, seg_pos={0 0}, upload=509234]: Done uploading piece. err: <nil> Dec 29 20:23:30.958595034 (580h54m30.829678069s) [seg_pos={0 0}, upload=509234]: Segment reached optimal threshold of 13 pieces. Dec 29 20:23:30.959013401 (580h54m30.830096476s) [noise=true , address=145.79.227.35:4481 , piece_id=IUATLJFBJ5HG3H4W44EFZCRJ7U5V3HXJHBQRTMRGYWGAEVQXQX6Q, seg_pos={0 0}, upload=509234]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bRK8WWmAnWUwzCfHu8JtZgrruYSJCvhnsYTKcaLccJcv5RQjp): protocol: expected piece hash; context canceled; EOF Dec 29 20:23:30.959036535 (580h54m30.8301196s) [noise=true , address=145.79.227.35:4481 , piece_id=IUATLJFBJ5HG3H4W44EFZCRJ7U5V3HXJHBQRTMRGYWGAEVQXQX6Q, seg_pos={0 0}, upload=509234]: Long tail cancellation due to optimal threshold Dec 29 20:23:30.959062274 (580h54m30.830145309s) [noise=true , address=45.91.116.11:4119 , piece_id=OL6RAER4DLQDSIQFBPQACL36ZAY6YXFCTW3MVELQT44M6MW665LQ, seg_pos={0 0}, upload=509234]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm): protocol: expected piece hash; context canceled; EOF Dec 29 20:23:30.959070019 (580h54m30.830153054s) [noise=true , address=45.91.116.11:4119 , piece_id=OL6RAER4DLQDSIQFBPQACL36ZAY6YXFCTW3MVELQT44M6MW665LQ, seg_pos={0 0}, upload=509234]: Long tail cancellation due to optimal threshold Dec 29 20:23:30.959092252 (580h54m30.830175297s) [noise=true , address=45.91.116.8:4045 , piece_id=46CTNCF4EIA4QSWQRCGTFZRHDOAMFZG5VV3I2WBZYJE7YABX4HEQ, seg_pos={0 0}, upload=509234]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9): protocol: expected piece hash; context canceled; EOF Dec 29 20:23:30.959097441 (580h54m30.830180476s) [noise=true , address=45.91.116.8:4045 , piece_id=46CTNCF4EIA4QSWQRCGTFZRHDOAMFZG5VV3I2WBZYJE7YABX4HEQ, seg_pos={0 0}, upload=509234]: Long tail cancellation due to optimal threshold Dec 29 20:23:30.959117088 (580h54m30.830200123s) [seg_pos={0 0}, upload=509234]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 20:23:30.959186390 (580h54m30.830269425s) [upload=509234]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 20:23:30.959191369 (580h54m30.830274415s) [upload=509234]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 20:23:31.263824068 (580h54m31.134907103s) [upload=509234]: Done uploading segments... Dec 29 20:23:31.263840989 (580h54m31.134924024s) [upload=509234]: Upload finished. err: <nil> Duration: 6.015340047s core uplink us-select-1
checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing) antiblip: 1 out of 60 is failed (threshold=6) [..................X.........................................] duration: 255.521261ms remote: "tcp:207.211.208.129:443" target: https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt trace: https://jaeger.dev.storj.io/trace/4679501fce62ca7 FAIL (2025-12-29 20:25:46 UTC) orbiter warning: Get "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt": context deadline exceeded Duration: 255.551198ms edge linksharing bgp eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [.X....X..X.X....X.......X.....................X......X.X.X..] duration: 6.232584931s trace: https://jaeger.dev.storj.io/trace/69a6314bd3cfcdea FAIL (2025-12-29 20:25:20 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, Piece ID: MY4MH5QULSN3DBEGRFO75W2VHDANQNOHM2F6JNXIQUJXEGCL2J2A) context deadline exceeded; piecestore: (Node ID: 12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, Piece ID: 57WOZLBFS3KLYOTUM34EQ4RHBEOZEZP6BOWUMD2WK2QHDKQS257A) context deadline exceeded; piecestore: (Node ID: 1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, Piece ID: TLQONEXNWGMCRKW2AIZ25JK46DBEYPJGDIFYSQHIYEGHUDYCMG4A) context deadline exceeded ======LOGS====== Dec 29 20:23:28.415883495 (580h54m28.28696653s) [upload=509234]: Starting upload Dec 29 20:23:28.415951074 (580h54m28.287034109s) [upload=509234]: Done starting upload Dec 29 20:23:28.416064840 (580h54m28.287147885s) [upload=509234]: Uploading segments... Dec 29 20:23:28.416264120 (580h54m28.287347165s) [upload=509234]: Got next segment. Inline: false Dec 29 20:23:28.416310388 (580h54m28.287393423s) [upload=509234]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 20:23:28.416315598 (580h54m28.287398634s) [upload=509234]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 20:23:28.660976111 (580h54m28.532059146s) [seg_pos={0 0}, upload=509234]: Begin upload segment... Dec 29 20:23:28.661026877 (580h54m28.532109922s) [seg_pos={0 0}, upload=509234]: Done begin upload segment. Dec 29 20:23:28.661068466 (580h54m28.532151501s) [noise=true , address=145.79.227.48:4303 , piece_id=KC5NQZIPPNPLQKZL7EYL7RL5Q3HLL4GQQ6S42AHTDENJJ6ZX6ZJA, seg_pos={0 0}, upload=509234]: Uploading piece... Dec 29 20:23:28.661072965 (580h54m28.532156s) [noise=true , address=145.79.227.42:4123 , piece_id=OD5XMSISHKBNLSQNAMV3ZCLBFMRLM7CGS7XKPC2ZU2J5R5AD6GVQ, seg_pos={0 0}, upload=509234]: Uploading piece... Dec 29 20:23:28.661729705 (580h54m28.53281274s) [noise=true , address=145.79.227.39:4014 , piece_id=J2YF6RILHWJI7JLAWCYN4WXBRQMENM63WLRTKNV4H5S3ALX4Q5VQ, seg_pos={0 0}, upload=509234]: Uploading piece... Dec 29 20:23:28.662080493 (580h54m28.533163528s) [noise=true , address=45.91.116.8:4045 , piece_id=46CTNCF4EIA4QSWQRCGTFZRHDOAMFZG5VV3I2WBZYJE7YABX4HEQ, seg_pos={0 0}, upload=509234]: Uploading piece... Dec 29 20:23:28.662340749 (580h54m28.533423784s) [noise=true , address=77.90.31.11:4554 , piece_id=TLQONEXNWGMCRKW2AIZ25JK46DBEYPJGDIFYSQHIYEGHUDYCMG4A, seg_pos={0 0}, upload=509234]: Uploading piece... Dec 29 20:23:28.662600053 (580h54m28.533683098s) [noise=true , address=145.79.227.4:4187 , piece_id=FGSQSSM4EO73VDOYM5AJD2XZSYVU5LWUCOVOG3GRZGRCDUODSLDA, seg_pos={0 0}, upload=509234]: Uploading piece... Dec 29 20:23:28.664093767 (580h54m28.535176802s) [noise=true , address=145.79.227.47:4266 , piece_id=O3QWUUL63R42OY2CBZPPCIPLEMUTE7HJRYHHBOOEOUZU2NQZXXKQ, seg_pos={0 0}, upload=509234]: Uploading piece... Dec 29 20:23:28.664605071 (580h54m28.535688106s) [noise=true , address=145.79.227.35:4481 , piece_id=IUATLJFBJ5HG3H4W44EFZCRJ7U5V3HXJHBQRTMRGYWGAEVQXQX6Q, seg_pos={0 0}, upload=509234]: Uploading piece... Dec 29 20:23:28.665273063 (580h54m28.536356108s) [noise=true , address=77.90.31.21:4878 , piece_id=T6ZKRL26GO7CGZ23I4N56F66YHWUM6XJ6D3AI6VFICGAH6GNICJQ, seg_pos={0 0}, upload=509234]: Uploading piece... Dec 29 20:23:28.665755532 (580h54m28.536838577s) [noise=true , address=45.91.116.6:4974 , piece_id=MY4MH5QULSN3DBEGRFO75W2VHDANQNOHM2F6JNXIQUJXEGCL2J2A, seg_pos={0 0}, upload=509234]: Uploading piece... Dec 29 20:23:28.667333967 (580h54m28.538417012s) [noise=true , address=45.91.116.11:4119 , piece_id=OL6RAER4DLQDSIQFBPQACL36ZAY6YXFCTW3MVELQT44M6MW665LQ, seg_pos={0 0}, upload=509234]: Uploading piece... Dec 29 20:23:28.668065620 (580h54m28.539148665s) [noise=true , address=145.79.227.16:4912 , piece_id=MTCEEL6SYO6DXOXXRNDN243LSYSOFVBLTMMGPFJOBZCJDCEONEPA, seg_pos={0 0}, upload=509234]: Uploading piece... Dec 29 20:23:28.669481887 (580h54m28.540564922s) [noise=true , address=145.79.227.7:4293 , piece_id=LX36RRBBW2YRBQJDRQ4V3FA7TSNI5BVRZW7CCSYJNWDRTCRD5KTA, seg_pos={0 0}, upload=509234]: Uploading piece... Dec 29 20:23:28.670629553 (580h54m28.541712588s) [noise=true , address=145.79.227.41:4085 , piece_id=O26CLJH4ICQNMHBTENAJLRNLDFLHA3CV2RMNGK32IZPWEER6DY2Q, seg_pos={0 0}, upload=509234]: Uploading piece... Dec 29 20:23:28.671866527 (580h54m28.542949562s) [noise=true , address=45.91.116.10:4082 , piece_id=N6HSAF6XCLGUZXDNGBHCGB3CSMQMBNETCFW7CHU7LJGTZ4JJLREA, seg_pos={0 0}, upload=509234]: Uploading piece... Dec 29 20:23:28.672948639 (580h54m28.544031674s) [noise=true , address=45.91.116.12:4226 , piece_id=57WOZLBFS3KLYOTUM34EQ4RHBEOZEZP6BOWUMD2WK2QHDKQS257A, seg_pos={0 0}, upload=509234]: Uploading piece... Dec 29 20:23:30.355599227 (580h54m30.226682262s) [upload=509234]: Next returned nil segment Dec 29 20:23:30.355633262 (580h54m30.226716307s) [upload=509234]: Waiting for error group managing segments... Dec 29 20:23:30.642553302 (580h54m30.513636337s) [noise=true , address=145.79.227.42:4123 , piece_id=OD5XMSISHKBNLSQNAMV3ZCLBFMRLM7CGS7XKPC2ZU2J5R5AD6GVQ, seg_pos={0 0}, upload=509234]: Done uploading piece. err: <nil> Dec 29 20:23:30.647195511 (580h54m30.518278546s) [noise=true , address=145.79.227.4:4187 , piece_id=FGSQSSM4EO73VDOYM5AJD2XZSYVU5LWUCOVOG3GRZGRCDUODSLDA, seg_pos={0 0}, upload=509234]: Done uploading piece. err: <nil> Dec 29 20:23:30.661620258 (580h54m30.532703303s) [noise=true , address=145.79.227.48:4303 , piece_id=KC5NQZIPPNPLQKZL7EYL7RL5Q3HLL4GQQ6S42AHTDENJJ6ZX6ZJA, seg_pos={0 0}, upload=509234]: Done uploading piece. err: <nil> Dec 29 20:23:30.664794151 (580h54m30.535877196s) [noise=true , address=145.79.227.41:4085 , piece_id=O26CLJH4ICQNMHBTENAJLRNLDFLHA3CV2RMNGK32IZPWEER6DY2Q, seg_pos={0 0}, upload=509234]: Done uploading piece. err: <nil> Dec 29 20:23:30.677472743 (580h54m30.548555788s) [noise=true , address=145.79.227.7:4293 , piece_id=LX36RRBBW2YRBQJDRQ4V3FA7TSNI5BVRZW7CCSYJNWDRTCRD5KTA, seg_pos={0 0}, upload=509234]: Done uploading piece. err: <nil> Dec 29 20:23:30.679803722 (580h54m30.550886767s) [noise=true , address=145.79.227.16:4912 , piece_id=MTCEEL6SYO6DXOXXRNDN243LSYSOFVBLTMMGPFJOBZCJDCEONEPA, seg_pos={0 0}, upload=509234]: Done uploading piece. err: <nil> Dec 29 20:23:30.689244619 (580h54m30.560327654s) [noise=true , address=145.79.227.39:4014 , piece_id=J2YF6RILHWJI7JLAWCYN4WXBRQMENM63WLRTKNV4H5S3ALX4Q5VQ, seg_pos={0 0}, upload=509234]: Done uploading piece. err: <nil> Dec 29 20:23:30.700784082 (580h54m30.571867117s) [noise=true , address=145.79.227.47:4266 , piece_id=O3QWUUL63R42OY2CBZPPCIPLEMUTE7HJRYHHBOOEOUZU2NQZXXKQ, seg_pos={0 0}, upload=509234]: Done uploading piece. err: <nil> Dec 29 20:23:30.847297316 (580h54m30.718380351s) [noise=true , address=77.90.31.11:4554 , piece_id=TLQONEXNWGMCRKW2AIZ25JK46DBEYPJGDIFYSQHIYEGHUDYCMG4A, seg_pos={0 0}, upload=509234]: Done uploading piece. err: <nil> Dec 29 20:23:30.920591620 (580h54m30.791674655s) [noise=true , address=45.91.116.10:4082 , piece_id=N6HSAF6XCLGUZXDNGBHCGB3CSMQMBNETCFW7CHU7LJGTZ4JJLREA, seg_pos={0 0}, upload=509234]: Done uploading piece. err: <nil> Dec 29 20:23:30.926928366 (580h54m30.798011401s) [noise=true , address=45.91.116.12:4226 , piece_id=57WOZLBFS3KLYOTUM34EQ4RHBEOZEZP6BOWUMD2WK2QHDKQS257A, seg_pos={0 0}, upload=509234]: Done uploading piece. err: <nil> Dec 29 20:23:30.945981767 (580h54m30.817064812s) [noise=true , address=45.91.116.6:4974 , piece_id=MY4MH5QULSN3DBEGRFO75W2VHDANQNOHM2F6JNXIQUJXEGCL2J2A, seg_pos={0 0}, upload=509234]: Done uploading piece. err: <nil> Dec 29 20:23:30.958572291 (580h54m30.829655326s) [noise=true , address=77.90.31.21:4878 , piece_id=T6ZKRL26GO7CGZ23I4N56F66YHWUM6XJ6D3AI6VFICGAH6GNICJQ, seg_pos={0 0}, upload=509234]: Done uploading piece. err: <nil> Dec 29 20:23:30.958595034 (580h54m30.829678069s) [seg_pos={0 0}, upload=509234]: Segment reached optimal threshold of 13 pieces. Dec 29 20:23:30.959013401 (580h54m30.830096476s) [noise=true , address=145.79.227.35:4481 , piece_id=IUATLJFBJ5HG3H4W44EFZCRJ7U5V3HXJHBQRTMRGYWGAEVQXQX6Q, seg_pos={0 0}, upload=509234]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bRK8WWmAnWUwzCfHu8JtZgrruYSJCvhnsYTKcaLccJcv5RQjp): protocol: expected piece hash; context canceled; EOF Dec 29 20:23:30.959036535 (580h54m30.8301196s) [noise=true , address=145.79.227.35:4481 , piece_id=IUATLJFBJ5HG3H4W44EFZCRJ7U5V3HXJHBQRTMRGYWGAEVQXQX6Q, seg_pos={0 0}, upload=509234]: Long tail cancellation due to optimal threshold Dec 29 20:23:30.959062274 (580h54m30.830145309s) [noise=true , address=45.91.116.11:4119 , piece_id=OL6RAER4DLQDSIQFBPQACL36ZAY6YXFCTW3MVELQT44M6MW665LQ, seg_pos={0 0}, upload=509234]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm): protocol: expected piece hash; context canceled; EOF Dec 29 20:23:30.959070019 (580h54m30.830153054s) [noise=true , address=45.91.116.11:4119 , piece_id=OL6RAER4DLQDSIQFBPQACL36ZAY6YXFCTW3MVELQT44M6MW665LQ, seg_pos={0 0}, upload=509234]: Long tail cancellation due to optimal threshold Dec 29 20:23:30.959092252 (580h54m30.830175297s) [noise=true , address=45.91.116.8:4045 , piece_id=46CTNCF4EIA4QSWQRCGTFZRHDOAMFZG5VV3I2WBZYJE7YABX4HEQ, seg_pos={0 0}, upload=509234]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9): protocol: expected piece hash; context canceled; EOF Dec 29 20:23:30.959097441 (580h54m30.830180476s) [noise=true , address=45.91.116.8:4045 , piece_id=46CTNCF4EIA4QSWQRCGTFZRHDOAMFZG5VV3I2WBZYJE7YABX4HEQ, seg_pos={0 0}, upload=509234]: Long tail cancellation due to optimal threshold Dec 29 20:23:30.959117088 (580h54m30.830200123s) [seg_pos={0 0}, upload=509234]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 20:23:30.959186390 (580h54m30.830269425s) [upload=509234]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 20:23:30.959191369 (580h54m30.830274415s) [upload=509234]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 20:23:31.263824068 (580h54m31.134907103s) [upload=509234]: Done uploading segments... Dec 29 20:23:31.263840989 (580h54m31.134924024s) [upload=509234]: Upload finished. err: <nil> Duration: 6.23261542s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [X.X....X..X.X....X.......X.....................X......X.X.X.] duration: 6.079117221s trace: https://jaeger.dev.storj.io/trace/69a6314bd3cfcdea FAIL (2025-12-29 20:24:46 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, Piece ID: MY4MH5QULSN3DBEGRFO75W2VHDANQNOHM2F6JNXIQUJXEGCL2J2A) context deadline exceeded; piecestore: (Node ID: 12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, Piece ID: 57WOZLBFS3KLYOTUM34EQ4RHBEOZEZP6BOWUMD2WK2QHDKQS257A) context deadline exceeded; piecestore: (Node ID: 1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, Piece ID: TLQONEXNWGMCRKW2AIZ25JK46DBEYPJGDIFYSQHIYEGHUDYCMG4A) context deadline exceeded ======LOGS====== Dec 29 20:23:28.415883495 (580h54m28.28696653s) [upload=509234]: Starting upload Dec 29 20:23:28.415951074 (580h54m28.287034109s) [upload=509234]: Done starting upload Dec 29 20:23:28.416064840 (580h54m28.287147885s) [upload=509234]: Uploading segments... Dec 29 20:23:28.416264120 (580h54m28.287347165s) [upload=509234]: Got next segment. Inline: false Dec 29 20:23:28.416310388 (580h54m28.287393423s) [upload=509234]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 20:23:28.416315598 (580h54m28.287398634s) [upload=509234]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 20:23:28.660976111 (580h54m28.532059146s) [seg_pos={0 0}, upload=509234]: Begin upload segment... Dec 29 20:23:28.661026877 (580h54m28.532109922s) [seg_pos={0 0}, upload=509234]: Done begin upload segment. Dec 29 20:23:28.661068466 (580h54m28.532151501s) [noise=true , address=145.79.227.48:4303 , piece_id=KC5NQZIPPNPLQKZL7EYL7RL5Q3HLL4GQQ6S42AHTDENJJ6ZX6ZJA, seg_pos={0 0}, upload=509234]: Uploading piece... Dec 29 20:23:28.661072965 (580h54m28.532156s) [noise=true , address=145.79.227.42:4123 , piece_id=OD5XMSISHKBNLSQNAMV3ZCLBFMRLM7CGS7XKPC2ZU2J5R5AD6GVQ, seg_pos={0 0}, upload=509234]: Uploading piece... Dec 29 20:23:28.661729705 (580h54m28.53281274s) [noise=true , address=145.79.227.39:4014 , piece_id=J2YF6RILHWJI7JLAWCYN4WXBRQMENM63WLRTKNV4H5S3ALX4Q5VQ, seg_pos={0 0}, upload=509234]: Uploading piece... Dec 29 20:23:28.662080493 (580h54m28.533163528s) [noise=true , address=45.91.116.8:4045 , piece_id=46CTNCF4EIA4QSWQRCGTFZRHDOAMFZG5VV3I2WBZYJE7YABX4HEQ, seg_pos={0 0}, upload=509234]: Uploading piece... Dec 29 20:23:28.662340749 (580h54m28.533423784s) [noise=true , address=77.90.31.11:4554 , piece_id=TLQONEXNWGMCRKW2AIZ25JK46DBEYPJGDIFYSQHIYEGHUDYCMG4A, seg_pos={0 0}, upload=509234]: Uploading piece... Dec 29 20:23:28.662600053 (580h54m28.533683098s) [noise=true , address=145.79.227.4:4187 , piece_id=FGSQSSM4EO73VDOYM5AJD2XZSYVU5LWUCOVOG3GRZGRCDUODSLDA, seg_pos={0 0}, upload=509234]: Uploading piece... Dec 29 20:23:28.664093767 (580h54m28.535176802s) [noise=true , address=145.79.227.47:4266 , piece_id=O3QWUUL63R42OY2CBZPPCIPLEMUTE7HJRYHHBOOEOUZU2NQZXXKQ, seg_pos={0 0}, upload=509234]: Uploading piece... Dec 29 20:23:28.664605071 (580h54m28.535688106s) [noise=true , address=145.79.227.35:4481 , piece_id=IUATLJFBJ5HG3H4W44EFZCRJ7U5V3HXJHBQRTMRGYWGAEVQXQX6Q, seg_pos={0 0}, upload=509234]: Uploading piece... Dec 29 20:23:28.665273063 (580h54m28.536356108s) [noise=true , address=77.90.31.21:4878 , piece_id=T6ZKRL26GO7CGZ23I4N56F66YHWUM6XJ6D3AI6VFICGAH6GNICJQ, seg_pos={0 0}, upload=509234]: Uploading piece... Dec 29 20:23:28.665755532 (580h54m28.536838577s) [noise=true , address=45.91.116.6:4974 , piece_id=MY4MH5QULSN3DBEGRFO75W2VHDANQNOHM2F6JNXIQUJXEGCL2J2A, seg_pos={0 0}, upload=509234]: Uploading piece... Dec 29 20:23:28.667333967 (580h54m28.538417012s) [noise=true , address=45.91.116.11:4119 , piece_id=OL6RAER4DLQDSIQFBPQACL36ZAY6YXFCTW3MVELQT44M6MW665LQ, seg_pos={0 0}, upload=509234]: Uploading piece... Dec 29 20:23:28.668065620 (580h54m28.539148665s) [noise=true , address=145.79.227.16:4912 , piece_id=MTCEEL6SYO6DXOXXRNDN243LSYSOFVBLTMMGPFJOBZCJDCEONEPA, seg_pos={0 0}, upload=509234]: Uploading piece... Dec 29 20:23:28.669481887 (580h54m28.540564922s) [noise=true , address=145.79.227.7:4293 , piece_id=LX36RRBBW2YRBQJDRQ4V3FA7TSNI5BVRZW7CCSYJNWDRTCRD5KTA, seg_pos={0 0}, upload=509234]: Uploading piece... Dec 29 20:23:28.670629553 (580h54m28.541712588s) [noise=true , address=145.79.227.41:4085 , piece_id=O26CLJH4ICQNMHBTENAJLRNLDFLHA3CV2RMNGK32IZPWEER6DY2Q, seg_pos={0 0}, upload=509234]: Uploading piece... Dec 29 20:23:28.671866527 (580h54m28.542949562s) [noise=true , address=45.91.116.10:4082 , piece_id=N6HSAF6XCLGUZXDNGBHCGB3CSMQMBNETCFW7CHU7LJGTZ4JJLREA, seg_pos={0 0}, upload=509234]: Uploading piece... Dec 29 20:23:28.672948639 (580h54m28.544031674s) [noise=true , address=45.91.116.12:4226 , piece_id=57WOZLBFS3KLYOTUM34EQ4RHBEOZEZP6BOWUMD2WK2QHDKQS257A, seg_pos={0 0}, upload=509234]: Uploading piece... Dec 29 20:23:30.355599227 (580h54m30.226682262s) [upload=509234]: Next returned nil segment Dec 29 20:23:30.355633262 (580h54m30.226716307s) [upload=509234]: Waiting for error group managing segments... Dec 29 20:23:30.642553302 (580h54m30.513636337s) [noise=true , address=145.79.227.42:4123 , piece_id=OD5XMSISHKBNLSQNAMV3ZCLBFMRLM7CGS7XKPC2ZU2J5R5AD6GVQ, seg_pos={0 0}, upload=509234]: Done uploading piece. err: <nil> Dec 29 20:23:30.647195511 (580h54m30.518278546s) [noise=true , address=145.79.227.4:4187 , piece_id=FGSQSSM4EO73VDOYM5AJD2XZSYVU5LWUCOVOG3GRZGRCDUODSLDA, seg_pos={0 0}, upload=509234]: Done uploading piece. err: <nil> Dec 29 20:23:30.661620258 (580h54m30.532703303s) [noise=true , address=145.79.227.48:4303 , piece_id=KC5NQZIPPNPLQKZL7EYL7RL5Q3HLL4GQQ6S42AHTDENJJ6ZX6ZJA, seg_pos={0 0}, upload=509234]: Done uploading piece. err: <nil> Dec 29 20:23:30.664794151 (580h54m30.535877196s) [noise=true , address=145.79.227.41:4085 , piece_id=O26CLJH4ICQNMHBTENAJLRNLDFLHA3CV2RMNGK32IZPWEER6DY2Q, seg_pos={0 0}, upload=509234]: Done uploading piece. err: <nil> Dec 29 20:23:30.677472743 (580h54m30.548555788s) [noise=true , address=145.79.227.7:4293 , piece_id=LX36RRBBW2YRBQJDRQ4V3FA7TSNI5BVRZW7CCSYJNWDRTCRD5KTA, seg_pos={0 0}, upload=509234]: Done uploading piece. err: <nil> Dec 29 20:23:30.679803722 (580h54m30.550886767s) [noise=true , address=145.79.227.16:4912 , piece_id=MTCEEL6SYO6DXOXXRNDN243LSYSOFVBLTMMGPFJOBZCJDCEONEPA, seg_pos={0 0}, upload=509234]: Done uploading piece. err: <nil> Dec 29 20:23:30.689244619 (580h54m30.560327654s) [noise=true , address=145.79.227.39:4014 , piece_id=J2YF6RILHWJI7JLAWCYN4WXBRQMENM63WLRTKNV4H5S3ALX4Q5VQ, seg_pos={0 0}, upload=509234]: Done uploading piece. err: <nil> Dec 29 20:23:30.700784082 (580h54m30.571867117s) [noise=true , address=145.79.227.47:4266 , piece_id=O3QWUUL63R42OY2CBZPPCIPLEMUTE7HJRYHHBOOEOUZU2NQZXXKQ, seg_pos={0 0}, upload=509234]: Done uploading piece. err: <nil> Dec 29 20:23:30.847297316 (580h54m30.718380351s) [noise=true , address=77.90.31.11:4554 , piece_id=TLQONEXNWGMCRKW2AIZ25JK46DBEYPJGDIFYSQHIYEGHUDYCMG4A, seg_pos={0 0}, upload=509234]: Done uploading piece. err: <nil> Dec 29 20:23:30.920591620 (580h54m30.791674655s) [noise=true , address=45.91.116.10:4082 , piece_id=N6HSAF6XCLGUZXDNGBHCGB3CSMQMBNETCFW7CHU7LJGTZ4JJLREA, seg_pos={0 0}, upload=509234]: Done uploading piece. err: <nil> Dec 29 20:23:30.926928366 (580h54m30.798011401s) [noise=true , address=45.91.116.12:4226 , piece_id=57WOZLBFS3KLYOTUM34EQ4RHBEOZEZP6BOWUMD2WK2QHDKQS257A, seg_pos={0 0}, upload=509234]: Done uploading piece. err: <nil> Dec 29 20:23:30.945981767 (580h54m30.817064812s) [noise=true , address=45.91.116.6:4974 , piece_id=MY4MH5QULSN3DBEGRFO75W2VHDANQNOHM2F6JNXIQUJXEGCL2J2A, seg_pos={0 0}, upload=509234]: Done uploading piece. err: <nil> Dec 29 20:23:30.958572291 (580h54m30.829655326s) [noise=true , address=77.90.31.21:4878 , piece_id=T6ZKRL26GO7CGZ23I4N56F66YHWUM6XJ6D3AI6VFICGAH6GNICJQ, seg_pos={0 0}, upload=509234]: Done uploading piece. err: <nil> Dec 29 20:23:30.958595034 (580h54m30.829678069s) [seg_pos={0 0}, upload=509234]: Segment reached optimal threshold of 13 pieces. Dec 29 20:23:30.959013401 (580h54m30.830096476s) [noise=true , address=145.79.227.35:4481 , piece_id=IUATLJFBJ5HG3H4W44EFZCRJ7U5V3HXJHBQRTMRGYWGAEVQXQX6Q, seg_pos={0 0}, upload=509234]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bRK8WWmAnWUwzCfHu8JtZgrruYSJCvhnsYTKcaLccJcv5RQjp): protocol: expected piece hash; context canceled; EOF Dec 29 20:23:30.959036535 (580h54m30.8301196s) [noise=true , address=145.79.227.35:4481 , piece_id=IUATLJFBJ5HG3H4W44EFZCRJ7U5V3HXJHBQRTMRGYWGAEVQXQX6Q, seg_pos={0 0}, upload=509234]: Long tail cancellation due to optimal threshold Dec 29 20:23:30.959062274 (580h54m30.830145309s) [noise=true , address=45.91.116.11:4119 , piece_id=OL6RAER4DLQDSIQFBPQACL36ZAY6YXFCTW3MVELQT44M6MW665LQ, seg_pos={0 0}, upload=509234]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm): protocol: expected piece hash; context canceled; EOF Dec 29 20:23:30.959070019 (580h54m30.830153054s) [noise=true , address=45.91.116.11:4119 , piece_id=OL6RAER4DLQDSIQFBPQACL36ZAY6YXFCTW3MVELQT44M6MW665LQ, seg_pos={0 0}, upload=509234]: Long tail cancellation due to optimal threshold Dec 29 20:23:30.959092252 (580h54m30.830175297s) [noise=true , address=45.91.116.8:4045 , piece_id=46CTNCF4EIA4QSWQRCGTFZRHDOAMFZG5VV3I2WBZYJE7YABX4HEQ, seg_pos={0 0}, upload=509234]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9): protocol: expected piece hash; context canceled; EOF Dec 29 20:23:30.959097441 (580h54m30.830180476s) [noise=true , address=45.91.116.8:4045 , piece_id=46CTNCF4EIA4QSWQRCGTFZRHDOAMFZG5VV3I2WBZYJE7YABX4HEQ, seg_pos={0 0}, upload=509234]: Long tail cancellation due to optimal threshold Dec 29 20:23:30.959117088 (580h54m30.830200123s) [seg_pos={0 0}, upload=509234]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 20:23:30.959186390 (580h54m30.830269425s) [upload=509234]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 20:23:30.959191369 (580h54m30.830274415s) [upload=509234]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 20:23:31.263824068 (580h54m31.134907103s) [upload=509234]: Done uploading segments... Dec 29 20:23:31.263840989 (580h54m31.134924024s) [upload=509234]: Upload finished. err: <nil> Duration: 6.079151746s core uplink us-select-1
checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing) antiblip: 1 out of 60 is failed (threshold=6) [...................X........................................] duration: 249.456323ms remote: "tcp:207.211.208.129:443" target: https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt trace: https://jaeger.dev.storj.io/trace/4679501fce62ca7 FAIL (2025-12-29 20:24:45 UTC) orbiter warning: Get "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt": context deadline exceeded Duration: 249.480118ms edge linksharing bgp eu1
checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing) antiblip: 1 out of 60 is failed (threshold=6) [....................X.......................................] duration: 256.073284ms remote: "tcp:207.211.208.129:443" target: https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt trace: https://jaeger.dev.storj.io/trace/4679501fce62ca7 FAIL (2025-12-29 20:24:13 UTC) orbiter warning: Get "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt": context deadline exceeded Duration: 256.096208ms edge linksharing bgp eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [.X.X....X..X.X....X.......X.....................X......X.X.X] duration: 30.003182999s trace: https://jaeger.dev.storj.io/trace/69a6314bd3cfcdea FAIL (2025-12-29 20:23:57 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, Piece ID: MY4MH5QULSN3DBEGRFO75W2VHDANQNOHM2F6JNXIQUJXEGCL2J2A) context deadline exceeded; piecestore: (Node ID: 12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, Piece ID: 57WOZLBFS3KLYOTUM34EQ4RHBEOZEZP6BOWUMD2WK2QHDKQS257A) context deadline exceeded; piecestore: (Node ID: 1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, Piece ID: TLQONEXNWGMCRKW2AIZ25JK46DBEYPJGDIFYSQHIYEGHUDYCMG4A) context deadline exceeded ======LOGS====== Dec 29 20:23:28.415883495 (580h54m28.28696653s) [upload=509234]: Starting upload Dec 29 20:23:28.415951074 (580h54m28.287034109s) [upload=509234]: Done starting upload Dec 29 20:23:28.416064840 (580h54m28.287147885s) [upload=509234]: Uploading segments... Dec 29 20:23:28.416264120 (580h54m28.287347165s) [upload=509234]: Got next segment. Inline: false Dec 29 20:23:28.416310388 (580h54m28.287393423s) [upload=509234]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 20:23:28.416315598 (580h54m28.287398634s) [upload=509234]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 20:23:28.660976111 (580h54m28.532059146s) [seg_pos={0 0}, upload=509234]: Begin upload segment... Dec 29 20:23:28.661026877 (580h54m28.532109922s) [seg_pos={0 0}, upload=509234]: Done begin upload segment. Dec 29 20:23:28.661068466 (580h54m28.532151501s) [noise=true , address=145.79.227.48:4303 , piece_id=KC5NQZIPPNPLQKZL7EYL7RL5Q3HLL4GQQ6S42AHTDENJJ6ZX6ZJA, seg_pos={0 0}, upload=509234]: Uploading piece... Dec 29 20:23:28.661072965 (580h54m28.532156s) [noise=true , address=145.79.227.42:4123 , piece_id=OD5XMSISHKBNLSQNAMV3ZCLBFMRLM7CGS7XKPC2ZU2J5R5AD6GVQ, seg_pos={0 0}, upload=509234]: Uploading piece... Dec 29 20:23:28.661729705 (580h54m28.53281274s) [noise=true , address=145.79.227.39:4014 , piece_id=J2YF6RILHWJI7JLAWCYN4WXBRQMENM63WLRTKNV4H5S3ALX4Q5VQ, seg_pos={0 0}, upload=509234]: Uploading piece... Dec 29 20:23:28.662080493 (580h54m28.533163528s) [noise=true , address=45.91.116.8:4045 , piece_id=46CTNCF4EIA4QSWQRCGTFZRHDOAMFZG5VV3I2WBZYJE7YABX4HEQ, seg_pos={0 0}, upload=509234]: Uploading piece... Dec 29 20:23:28.662340749 (580h54m28.533423784s) [noise=true , address=77.90.31.11:4554 , piece_id=TLQONEXNWGMCRKW2AIZ25JK46DBEYPJGDIFYSQHIYEGHUDYCMG4A, seg_pos={0 0}, upload=509234]: Uploading piece... Dec 29 20:23:28.662600053 (580h54m28.533683098s) [noise=true , address=145.79.227.4:4187 , piece_id=FGSQSSM4EO73VDOYM5AJD2XZSYVU5LWUCOVOG3GRZGRCDUODSLDA, seg_pos={0 0}, upload=509234]: Uploading piece... Dec 29 20:23:28.664093767 (580h54m28.535176802s) [noise=true , address=145.79.227.47:4266 , piece_id=O3QWUUL63R42OY2CBZPPCIPLEMUTE7HJRYHHBOOEOUZU2NQZXXKQ, seg_pos={0 0}, upload=509234]: Uploading piece... Dec 29 20:23:28.664605071 (580h54m28.535688106s) [noise=true , address=145.79.227.35:4481 , piece_id=IUATLJFBJ5HG3H4W44EFZCRJ7U5V3HXJHBQRTMRGYWGAEVQXQX6Q, seg_pos={0 0}, upload=509234]: Uploading piece... Dec 29 20:23:28.665273063 (580h54m28.536356108s) [noise=true , address=77.90.31.21:4878 , piece_id=T6ZKRL26GO7CGZ23I4N56F66YHWUM6XJ6D3AI6VFICGAH6GNICJQ, seg_pos={0 0}, upload=509234]: Uploading piece... Dec 29 20:23:28.665755532 (580h54m28.536838577s) [noise=true , address=45.91.116.6:4974 , piece_id=MY4MH5QULSN3DBEGRFO75W2VHDANQNOHM2F6JNXIQUJXEGCL2J2A, seg_pos={0 0}, upload=509234]: Uploading piece... Dec 29 20:23:28.667333967 (580h54m28.538417012s) [noise=true , address=45.91.116.11:4119 , piece_id=OL6RAER4DLQDSIQFBPQACL36ZAY6YXFCTW3MVELQT44M6MW665LQ, seg_pos={0 0}, upload=509234]: Uploading piece... Dec 29 20:23:28.668065620 (580h54m28.539148665s) [noise=true , address=145.79.227.16:4912 , piece_id=MTCEEL6SYO6DXOXXRNDN243LSYSOFVBLTMMGPFJOBZCJDCEONEPA, seg_pos={0 0}, upload=509234]: Uploading piece... Dec 29 20:23:28.669481887 (580h54m28.540564922s) [noise=true , address=145.79.227.7:4293 , piece_id=LX36RRBBW2YRBQJDRQ4V3FA7TSNI5BVRZW7CCSYJNWDRTCRD5KTA, seg_pos={0 0}, upload=509234]: Uploading piece... Dec 29 20:23:28.670629553 (580h54m28.541712588s) [noise=true , address=145.79.227.41:4085 , piece_id=O26CLJH4ICQNMHBTENAJLRNLDFLHA3CV2RMNGK32IZPWEER6DY2Q, seg_pos={0 0}, upload=509234]: Uploading piece... Dec 29 20:23:28.671866527 (580h54m28.542949562s) [noise=true , address=45.91.116.10:4082 , piece_id=N6HSAF6XCLGUZXDNGBHCGB3CSMQMBNETCFW7CHU7LJGTZ4JJLREA, seg_pos={0 0}, upload=509234]: Uploading piece... Dec 29 20:23:28.672948639 (580h54m28.544031674s) [noise=true , address=45.91.116.12:4226 , piece_id=57WOZLBFS3KLYOTUM34EQ4RHBEOZEZP6BOWUMD2WK2QHDKQS257A, seg_pos={0 0}, upload=509234]: Uploading piece... Dec 29 20:23:30.355599227 (580h54m30.226682262s) [upload=509234]: Next returned nil segment Dec 29 20:23:30.355633262 (580h54m30.226716307s) [upload=509234]: Waiting for error group managing segments... Dec 29 20:23:30.642553302 (580h54m30.513636337s) [noise=true , address=145.79.227.42:4123 , piece_id=OD5XMSISHKBNLSQNAMV3ZCLBFMRLM7CGS7XKPC2ZU2J5R5AD6GVQ, seg_pos={0 0}, upload=509234]: Done uploading piece. err: <nil> Dec 29 20:23:30.647195511 (580h54m30.518278546s) [noise=true , address=145.79.227.4:4187 , piece_id=FGSQSSM4EO73VDOYM5AJD2XZSYVU5LWUCOVOG3GRZGRCDUODSLDA, seg_pos={0 0}, upload=509234]: Done uploading piece. err: <nil> Dec 29 20:23:30.661620258 (580h54m30.532703303s) [noise=true , address=145.79.227.48:4303 , piece_id=KC5NQZIPPNPLQKZL7EYL7RL5Q3HLL4GQQ6S42AHTDENJJ6ZX6ZJA, seg_pos={0 0}, upload=509234]: Done uploading piece. err: <nil> Dec 29 20:23:30.664794151 (580h54m30.535877196s) [noise=true , address=145.79.227.41:4085 , piece_id=O26CLJH4ICQNMHBTENAJLRNLDFLHA3CV2RMNGK32IZPWEER6DY2Q, seg_pos={0 0}, upload=509234]: Done uploading piece. err: <nil> Dec 29 20:23:30.677472743 (580h54m30.548555788s) [noise=true , address=145.79.227.7:4293 , piece_id=LX36RRBBW2YRBQJDRQ4V3FA7TSNI5BVRZW7CCSYJNWDRTCRD5KTA, seg_pos={0 0}, upload=509234]: Done uploading piece. err: <nil> Dec 29 20:23:30.679803722 (580h54m30.550886767s) [noise=true , address=145.79.227.16:4912 , piece_id=MTCEEL6SYO6DXOXXRNDN243LSYSOFVBLTMMGPFJOBZCJDCEONEPA, seg_pos={0 0}, upload=509234]: Done uploading piece. err: <nil> Dec 29 20:23:30.689244619 (580h54m30.560327654s) [noise=true , address=145.79.227.39:4014 , piece_id=J2YF6RILHWJI7JLAWCYN4WXBRQMENM63WLRTKNV4H5S3ALX4Q5VQ, seg_pos={0 0}, upload=509234]: Done uploading piece. err: <nil> Dec 29 20:23:30.700784082 (580h54m30.571867117s) [noise=true , address=145.79.227.47:4266 , piece_id=O3QWUUL63R42OY2CBZPPCIPLEMUTE7HJRYHHBOOEOUZU2NQZXXKQ, seg_pos={0 0}, upload=509234]: Done uploading piece. err: <nil> Dec 29 20:23:30.847297316 (580h54m30.718380351s) [noise=true , address=77.90.31.11:4554 , piece_id=TLQONEXNWGMCRKW2AIZ25JK46DBEYPJGDIFYSQHIYEGHUDYCMG4A, seg_pos={0 0}, upload=509234]: Done uploading piece. err: <nil> Dec 29 20:23:30.920591620 (580h54m30.791674655s) [noise=true , address=45.91.116.10:4082 , piece_id=N6HSAF6XCLGUZXDNGBHCGB3CSMQMBNETCFW7CHU7LJGTZ4JJLREA, seg_pos={0 0}, upload=509234]: Done uploading piece. err: <nil> Dec 29 20:23:30.926928366 (580h54m30.798011401s) [noise=true , address=45.91.116.12:4226 , piece_id=57WOZLBFS3KLYOTUM34EQ4RHBEOZEZP6BOWUMD2WK2QHDKQS257A, seg_pos={0 0}, upload=509234]: Done uploading piece. err: <nil> Dec 29 20:23:30.945981767 (580h54m30.817064812s) [noise=true , address=45.91.116.6:4974 , piece_id=MY4MH5QULSN3DBEGRFO75W2VHDANQNOHM2F6JNXIQUJXEGCL2J2A, seg_pos={0 0}, upload=509234]: Done uploading piece. err: <nil> Dec 29 20:23:30.958572291 (580h54m30.829655326s) [noise=true , address=77.90.31.21:4878 , piece_id=T6ZKRL26GO7CGZ23I4N56F66YHWUM6XJ6D3AI6VFICGAH6GNICJQ, seg_pos={0 0}, upload=509234]: Done uploading piece. err: <nil> Dec 29 20:23:30.958595034 (580h54m30.829678069s) [seg_pos={0 0}, upload=509234]: Segment reached optimal threshold of 13 pieces. Dec 29 20:23:30.959013401 (580h54m30.830096476s) [noise=true , address=145.79.227.35:4481 , piece_id=IUATLJFBJ5HG3H4W44EFZCRJ7U5V3HXJHBQRTMRGYWGAEVQXQX6Q, seg_pos={0 0}, upload=509234]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bRK8WWmAnWUwzCfHu8JtZgrruYSJCvhnsYTKcaLccJcv5RQjp): protocol: expected piece hash; context canceled; EOF Dec 29 20:23:30.959036535 (580h54m30.8301196s) [noise=true , address=145.79.227.35:4481 , piece_id=IUATLJFBJ5HG3H4W44EFZCRJ7U5V3HXJHBQRTMRGYWGAEVQXQX6Q, seg_pos={0 0}, upload=509234]: Long tail cancellation due to optimal threshold Dec 29 20:23:30.959062274 (580h54m30.830145309s) [noise=true , address=45.91.116.11:4119 , piece_id=OL6RAER4DLQDSIQFBPQACL36ZAY6YXFCTW3MVELQT44M6MW665LQ, seg_pos={0 0}, upload=509234]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm): protocol: expected piece hash; context canceled; EOF Dec 29 20:23:30.959070019 (580h54m30.830153054s) [noise=true , address=45.91.116.11:4119 , piece_id=OL6RAER4DLQDSIQFBPQACL36ZAY6YXFCTW3MVELQT44M6MW665LQ, seg_pos={0 0}, upload=509234]: Long tail cancellation due to optimal threshold Dec 29 20:23:30.959092252 (580h54m30.830175297s) [noise=true , address=45.91.116.8:4045 , piece_id=46CTNCF4EIA4QSWQRCGTFZRHDOAMFZG5VV3I2WBZYJE7YABX4HEQ, seg_pos={0 0}, upload=509234]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9): protocol: expected piece hash; context canceled; EOF Dec 29 20:23:30.959097441 (580h54m30.830180476s) [noise=true , address=45.91.116.8:4045 , piece_id=46CTNCF4EIA4QSWQRCGTFZRHDOAMFZG5VV3I2WBZYJE7YABX4HEQ, seg_pos={0 0}, upload=509234]: Long tail cancellation due to optimal threshold Dec 29 20:23:30.959117088 (580h54m30.830200123s) [seg_pos={0 0}, upload=509234]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 20:23:30.959186390 (580h54m30.830269425s) [upload=509234]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 20:23:30.959191369 (580h54m30.830274415s) [upload=509234]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 20:23:31.263824068 (580h54m31.134907103s) [upload=509234]: Done uploading segments... Dec 29 20:23:31.263840989 (580h54m31.134924024s) [upload=509234]: Upload finished. err: <nil> Duration: 30.003207214s core uplink us-select-1
checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing) antiblip: 1 out of 60 is failed (threshold=6) [.....................X......................................] duration: 248.306905ms remote: "tcp:207.211.208.129:443" target: https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt trace: https://jaeger.dev.storj.io/trace/4679501fce62ca7 FAIL (2025-12-29 20:23:41 UTC) orbiter warning: Get "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt": context deadline exceeded Duration: 248.331141ms edge linksharing bgp eu1
checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing) antiblip: 1 out of 60 is failed (threshold=6) [......................X.....................................] duration: 255.671979ms remote: "tcp:207.211.208.129:443" target: https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt trace: https://jaeger.dev.storj.io/trace/4679501fce62ca7 FAIL (2025-12-29 20:22:57 UTC) orbiter warning: Get "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt": context deadline exceeded Duration: 255.695623ms edge linksharing bgp eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [..X.X....X..X.X....X.......X.....................X......X.X.] duration: 7.104660522s trace: https://jaeger.dev.storj.io/trace/c15179e9e7380ef FAIL (2025-12-29 20:22:28 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, Piece ID: NNYXQBCIK276MMFFNNSF55445ND6ZWNDS2D2FZRI6I7FFN6DYO4A) context deadline exceeded; piecestore: (Node ID: 1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, Piece ID: FJC3VHIH6C2CGQRF3GCK2I4IKOEFBZVHEGGY5RWTW5WVPCE27BLA) context deadline exceeded ======LOGS====== Dec 29 20:21:42.795597421 (580h52m42.666680457s) [upload=509207]: Starting upload Dec 29 20:21:42.795671562 (580h52m42.666754598s) [upload=509207]: Done starting upload Dec 29 20:21:42.795919995 (580h52m42.66700304s) [upload=509207]: Uploading segments... Dec 29 20:21:42.795987424 (580h52m42.667070459s) [upload=509207]: Got next segment. Inline: false Dec 29 20:21:42.796015337 (580h52m42.667098372s) [upload=509207]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 20:21:42.796034063 (580h52m42.667117098s) [upload=509207]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 20:21:43.042610613 (580h52m42.913693648s) [seg_pos={0 0}, upload=509207]: Begin upload segment... Dec 29 20:21:43.042714491 (580h52m42.913797526s) [seg_pos={0 0}, upload=509207]: Done begin upload segment. Dec 29 20:21:43.042757964 (580h52m42.913841009s) [noise=true , address=145.79.227.2:4114 , piece_id=M2Y2S4FAQRCAFPQDQQX2JPCFOD5ODGXKKDCTODIJNWXCFGTG5DBA, seg_pos={0 0}, upload=509207]: Uploading piece... Dec 29 20:21:43.042773723 (580h52m42.913856768s) [noise=true , address=77.90.31.16:4736 , piece_id=QIYITDUU2XGY365LQVWLRAVOQB6HJ7DA4OVGY2NRSHEQRL57VI5A, seg_pos={0 0}, upload=509207]: Uploading piece... Dec 29 20:21:43.044124255 (580h52m42.9152073s) [noise=true , address=145.79.227.49:4337 , piece_id=7MLBIBNLWZNMBHKNPHT4P6TDXDE4NIEPHIPIOTPO3C6ULSX65BMQ, seg_pos={0 0}, upload=509207]: Uploading piece... Dec 29 20:21:43.044364412 (580h52m42.915447458s) [noise=true , address=45.91.116.7:4011 , piece_id=DDIKOM35SDUNGMSHPW2JJFIJ65BU6YWKMYJRXOGYMAHRUYS4GKNA, seg_pos={0 0}, upload=509207]: Uploading piece... Dec 29 20:21:43.044717134 (580h52m42.915800179s) [noise=true , address=45.91.116.11:4119 , piece_id=OBVWSCNGH4VME6SU63QSGEPC6T65WC6325B4JR2LHTEN5YPNYQKA, seg_pos={0 0}, upload=509207]: Uploading piece... Dec 29 20:21:43.044959135 (580h52m42.91604218s) [noise=true , address=45.91.116.5:4298 , piece_id=NNYXQBCIK276MMFFNNSF55445ND6ZWNDS2D2FZRI6I7FFN6DYO4A, seg_pos={0 0}, upload=509207]: Uploading piece... Dec 29 20:21:43.045166610 (580h52m42.916249655s) [noise=true , address=77.90.31.19:4842 , piece_id=FJC3VHIH6C2CGQRF3GCK2I4IKOEFBZVHEGGY5RWTW5WVPCE27BLA, seg_pos={0 0}, upload=509207]: Uploading piece... Dec 29 20:21:43.045604264 (580h52m42.916687299s) [noise=true , address=145.79.227.6:4257 , piece_id=ZUQK7DGTYNMCBOBEL3CB5IS2VPR4GQWHTYK5W23L2MHITQZJMCEQ, seg_pos={0 0}, upload=509207]: Uploading piece... Dec 29 20:21:43.045789246 (580h52m42.916872291s) [noise=true , address=145.79.227.11:4730 , piece_id=2ER4QY6EOV4V6N5KHFRE6RPC5NGS3UCWR37WSXX2DKLOOLCTI6XQ, seg_pos={0 0}, upload=509207]: Uploading piece... Dec 29 20:21:43.046506402 (580h52m42.917589447s) [noise=true , address=145.79.227.47:4266 , piece_id=K4RUZUJHMHOPKF4JAX54XFKNG66X6HKHD6FHGABMRDKAUMYGXH5A, seg_pos={0 0}, upload=509207]: Uploading piece... Dec 29 20:21:43.047358233 (580h52m42.918441278s) [noise=true , address=77.90.31.13:4629 , piece_id=ZFD3VXFJCRK6VUR2LWE4HR74OH5DSL2BGA2RBYDKRQH2LELSBTEA, seg_pos={0 0}, upload=509207]: Uploading piece... Dec 29 20:21:43.047721334 (580h52m42.918804379s) [noise=true , address=77.90.31.18:4808 , piece_id=XZBEJFFOCX6ZDX6FAKT6TZ4N6I44DRV6HU6BHPRFK34CEC4R722Q, seg_pos={0 0}, upload=509207]: Uploading piece... Dec 29 20:21:43.048010145 (580h52m42.91909319s) [noise=true , address=77.90.31.10:4519 , piece_id=AHUZFYFWDH54G4XWM2FW56UTRUCFZ6B2CFSXQEYZ67B2BBTIFY4Q, seg_pos={0 0}, upload=509207]: Uploading piece... Dec 29 20:21:43.048342157 (580h52m42.919425202s) [noise=true , address=145.79.227.14:4840 , piece_id=HLRLJWIF3CV2CKZGXYMLNBO2YTD6DOYDSRRLDSUQKMGXFLU2MIPA, seg_pos={0 0}, upload=509207]: Uploading piece... Dec 29 20:21:43.049040837 (580h52m42.920123882s) [noise=true , address=45.91.116.9:4262 , piece_id=QSGWHDQDGDBSF3GFQYQIQXUNDYGG3MC72QC4R52KBHFZAYXBOK4Q, seg_pos={0 0}, upload=509207]: Uploading piece... Dec 29 20:21:43.049783241 (580h52m42.920866276s) [noise=true , address=145.79.227.34:4906 , piece_id=UWDOJZ6SZ44CU3OMKCDDLHKUXHR6R3N77SZRC3SG4NVXE4G7XTAQ, seg_pos={0 0}, upload=509207]: Uploading piece... Dec 29 20:21:44.733752709 (580h52m44.604835754s) [upload=509207]: Next returned nil segment Dec 29 20:21:44.733773028 (580h52m44.604856063s) [upload=509207]: Waiting for error group managing segments... Dec 29 20:21:45.018632965 (580h52m44.88971601s) [noise=true , address=145.79.227.6:4257 , piece_id=ZUQK7DGTYNMCBOBEL3CB5IS2VPR4GQWHTYK5W23L2MHITQZJMCEQ, seg_pos={0 0}, upload=509207]: Done uploading piece. err: <nil> Dec 29 20:21:45.048385819 (580h52m44.919468854s) [noise=true , address=145.79.227.2:4114 , piece_id=M2Y2S4FAQRCAFPQDQQX2JPCFOD5ODGXKKDCTODIJNWXCFGTG5DBA, seg_pos={0 0}, upload=509207]: Done uploading piece. err: <nil> Dec 29 20:21:45.057250780 (580h52m44.928333815s) [noise=true , address=145.79.227.11:4730 , piece_id=2ER4QY6EOV4V6N5KHFRE6RPC5NGS3UCWR37WSXX2DKLOOLCTI6XQ, seg_pos={0 0}, upload=509207]: Done uploading piece. err: <nil> Dec 29 20:21:45.078675265 (580h52m44.949758311s) [noise=true , address=145.79.227.47:4266 , piece_id=K4RUZUJHMHOPKF4JAX54XFKNG66X6HKHD6FHGABMRDKAUMYGXH5A, seg_pos={0 0}, upload=509207]: Done uploading piece. err: <nil> Dec 29 20:21:45.157138352 (580h52m45.028221397s) [noise=true , address=145.79.227.14:4840 , piece_id=HLRLJWIF3CV2CKZGXYMLNBO2YTD6DOYDSRRLDSUQKMGXFLU2MIPA, seg_pos={0 0}, upload=509207]: Done uploading piece. err: <nil> Dec 29 20:21:45.225140328 (580h52m45.096223363s) [noise=true , address=77.90.31.19:4842 , piece_id=FJC3VHIH6C2CGQRF3GCK2I4IKOEFBZVHEGGY5RWTW5WVPCE27BLA, seg_pos={0 0}, upload=509207]: Done uploading piece. err: <nil> Dec 29 20:21:45.258788668 (580h52m45.129871703s) [noise=true , address=77.90.31.16:4736 , piece_id=QIYITDUU2XGY365LQVWLRAVOQB6HJ7DA4OVGY2NRSHEQRL57VI5A, seg_pos={0 0}, upload=509207]: Done uploading piece. err: <nil> Dec 29 20:21:45.276519862 (580h52m45.147602907s) [noise=true , address=45.91.116.9:4262 , piece_id=QSGWHDQDGDBSF3GFQYQIQXUNDYGG3MC72QC4R52KBHFZAYXBOK4Q, seg_pos={0 0}, upload=509207]: Done uploading piece. err: <nil> Dec 29 20:21:45.286221385 (580h52m45.15730442s) [noise=true , address=77.90.31.10:4519 , piece_id=AHUZFYFWDH54G4XWM2FW56UTRUCFZ6B2CFSXQEYZ67B2BBTIFY4Q, seg_pos={0 0}, upload=509207]: Done uploading piece. err: <nil> Dec 29 20:21:45.321639817 (580h52m45.192722862s) [noise=true , address=77.90.31.18:4808 , piece_id=XZBEJFFOCX6ZDX6FAKT6TZ4N6I44DRV6HU6BHPRFK34CEC4R722Q, seg_pos={0 0}, upload=509207]: Done uploading piece. err: <nil> Dec 29 20:21:45.335734296 (580h52m45.206817331s) [noise=true , address=45.91.116.11:4119 , piece_id=OBVWSCNGH4VME6SU63QSGEPC6T65WC6325B4JR2LHTEN5YPNYQKA, seg_pos={0 0}, upload=509207]: Done uploading piece. err: <nil> Dec 29 20:21:45.338167799 (580h52m45.209250834s) [noise=true , address=77.90.31.13:4629 , piece_id=ZFD3VXFJCRK6VUR2LWE4HR74OH5DSL2BGA2RBYDKRQH2LELSBTEA, seg_pos={0 0}, upload=509207]: Done uploading piece. err: <nil> Dec 29 20:21:45.370591519 (580h52m45.241674564s) [noise=true , address=45.91.116.5:4298 , piece_id=NNYXQBCIK276MMFFNNSF55445ND6ZWNDS2D2FZRI6I7FFN6DYO4A, seg_pos={0 0}, upload=509207]: Done uploading piece. err: <nil> Dec 29 20:21:45.370612739 (580h52m45.241695774s) [seg_pos={0 0}, upload=509207]: Segment reached optimal threshold of 13 pieces. Dec 29 20:21:45.370859700 (580h52m45.241942765s) [noise=true , address=145.79.227.34:4906 , piece_id=UWDOJZ6SZ44CU3OMKCDDLHKUXHR6R3N77SZRC3SG4NVXE4G7XTAQ, seg_pos={0 0}, upload=509207]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw): protocol: expected piece hash; context canceled; EOF Dec 29 20:21:45.370870681 (580h52m45.241953716s) [noise=true , address=145.79.227.34:4906 , piece_id=UWDOJZ6SZ44CU3OMKCDDLHKUXHR6R3N77SZRC3SG4NVXE4G7XTAQ, seg_pos={0 0}, upload=509207]: Long tail cancellation due to optimal threshold Dec 29 20:21:45.370910296 (580h52m45.241993341s) [noise=true , address=145.79.227.49:4337 , piece_id=7MLBIBNLWZNMBHKNPHT4P6TDXDE4NIEPHIPIOTPO3C6ULSX65BMQ, seg_pos={0 0}, upload=509207]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nTZFfzNCcVAjut7NXf9seifysJDfvXkS8aZSnXKfw6gh1TrjE): protocol: expected piece hash; context canceled; EOF Dec 29 20:21:45.370919233 (580h52m45.242002268s) [noise=true , address=145.79.227.49:4337 , piece_id=7MLBIBNLWZNMBHKNPHT4P6TDXDE4NIEPHIPIOTPO3C6ULSX65BMQ, seg_pos={0 0}, upload=509207]: Long tail cancellation due to optimal threshold Dec 29 20:21:45.371157677 (580h52m45.242240722s) [noise=true , address=45.91.116.7:4011 , piece_id=DDIKOM35SDUNGMSHPW2JJFIJ65BU6YWKMYJRXOGYMAHRUYS4GKNA, seg_pos={0 0}, upload=509207]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf): protocol: context canceled; context canceled Dec 29 20:21:45.371166624 (580h52m45.242249659s) [noise=true , address=45.91.116.7:4011 , piece_id=DDIKOM35SDUNGMSHPW2JJFIJ65BU6YWKMYJRXOGYMAHRUYS4GKNA, seg_pos={0 0}, upload=509207]: Long tail cancellation due to optimal threshold Dec 29 20:21:45.371187755 (580h52m45.2422708s) [seg_pos={0 0}, upload=509207]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 20:21:45.371268829 (580h52m45.242351874s) [upload=509207]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 20:21:45.371277245 (580h52m45.24236029s) [upload=509207]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 20:21:45.669010465 (580h52m45.5400935s) [upload=509207]: Done uploading segments... Dec 29 20:21:45.669034200 (580h52m45.540117235s) [upload=509207]: Upload finished. err: <nil> Duration: 7.104689448s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [...X.X....X..X.X....X.......X.....................X......X.X] duration: 30.00286616s trace: https://jaeger.dev.storj.io/trace/c15179e9e7380ef FAIL (2025-12-29 20:22:12 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, Piece ID: NNYXQBCIK276MMFFNNSF55445ND6ZWNDS2D2FZRI6I7FFN6DYO4A) context deadline exceeded; piecestore: (Node ID: 1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, Piece ID: FJC3VHIH6C2CGQRF3GCK2I4IKOEFBZVHEGGY5RWTW5WVPCE27BLA) context deadline exceeded ======LOGS====== Dec 29 20:21:42.795597421 (580h52m42.666680457s) [upload=509207]: Starting upload Dec 29 20:21:42.795671562 (580h52m42.666754598s) [upload=509207]: Done starting upload Dec 29 20:21:42.795919995 (580h52m42.66700304s) [upload=509207]: Uploading segments... Dec 29 20:21:42.795987424 (580h52m42.667070459s) [upload=509207]: Got next segment. Inline: false Dec 29 20:21:42.796015337 (580h52m42.667098372s) [upload=509207]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 20:21:42.796034063 (580h52m42.667117098s) [upload=509207]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 20:21:43.042610613 (580h52m42.913693648s) [seg_pos={0 0}, upload=509207]: Begin upload segment... Dec 29 20:21:43.042714491 (580h52m42.913797526s) [seg_pos={0 0}, upload=509207]: Done begin upload segment. Dec 29 20:21:43.042757964 (580h52m42.913841009s) [noise=true , address=145.79.227.2:4114 , piece_id=M2Y2S4FAQRCAFPQDQQX2JPCFOD5ODGXKKDCTODIJNWXCFGTG5DBA, seg_pos={0 0}, upload=509207]: Uploading piece... Dec 29 20:21:43.042773723 (580h52m42.913856768s) [noise=true , address=77.90.31.16:4736 , piece_id=QIYITDUU2XGY365LQVWLRAVOQB6HJ7DA4OVGY2NRSHEQRL57VI5A, seg_pos={0 0}, upload=509207]: Uploading piece... Dec 29 20:21:43.044124255 (580h52m42.9152073s) [noise=true , address=145.79.227.49:4337 , piece_id=7MLBIBNLWZNMBHKNPHT4P6TDXDE4NIEPHIPIOTPO3C6ULSX65BMQ, seg_pos={0 0}, upload=509207]: Uploading piece... Dec 29 20:21:43.044364412 (580h52m42.915447458s) [noise=true , address=45.91.116.7:4011 , piece_id=DDIKOM35SDUNGMSHPW2JJFIJ65BU6YWKMYJRXOGYMAHRUYS4GKNA, seg_pos={0 0}, upload=509207]: Uploading piece... Dec 29 20:21:43.044717134 (580h52m42.915800179s) [noise=true , address=45.91.116.11:4119 , piece_id=OBVWSCNGH4VME6SU63QSGEPC6T65WC6325B4JR2LHTEN5YPNYQKA, seg_pos={0 0}, upload=509207]: Uploading piece... Dec 29 20:21:43.044959135 (580h52m42.91604218s) [noise=true , address=45.91.116.5:4298 , piece_id=NNYXQBCIK276MMFFNNSF55445ND6ZWNDS2D2FZRI6I7FFN6DYO4A, seg_pos={0 0}, upload=509207]: Uploading piece... Dec 29 20:21:43.045166610 (580h52m42.916249655s) [noise=true , address=77.90.31.19:4842 , piece_id=FJC3VHIH6C2CGQRF3GCK2I4IKOEFBZVHEGGY5RWTW5WVPCE27BLA, seg_pos={0 0}, upload=509207]: Uploading piece... Dec 29 20:21:43.045604264 (580h52m42.916687299s) [noise=true , address=145.79.227.6:4257 , piece_id=ZUQK7DGTYNMCBOBEL3CB5IS2VPR4GQWHTYK5W23L2MHITQZJMCEQ, seg_pos={0 0}, upload=509207]: Uploading piece... Dec 29 20:21:43.045789246 (580h52m42.916872291s) [noise=true , address=145.79.227.11:4730 , piece_id=2ER4QY6EOV4V6N5KHFRE6RPC5NGS3UCWR37WSXX2DKLOOLCTI6XQ, seg_pos={0 0}, upload=509207]: Uploading piece... Dec 29 20:21:43.046506402 (580h52m42.917589447s) [noise=true , address=145.79.227.47:4266 , piece_id=K4RUZUJHMHOPKF4JAX54XFKNG66X6HKHD6FHGABMRDKAUMYGXH5A, seg_pos={0 0}, upload=509207]: Uploading piece... Dec 29 20:21:43.047358233 (580h52m42.918441278s) [noise=true , address=77.90.31.13:4629 , piece_id=ZFD3VXFJCRK6VUR2LWE4HR74OH5DSL2BGA2RBYDKRQH2LELSBTEA, seg_pos={0 0}, upload=509207]: Uploading piece... Dec 29 20:21:43.047721334 (580h52m42.918804379s) [noise=true , address=77.90.31.18:4808 , piece_id=XZBEJFFOCX6ZDX6FAKT6TZ4N6I44DRV6HU6BHPRFK34CEC4R722Q, seg_pos={0 0}, upload=509207]: Uploading piece... Dec 29 20:21:43.048010145 (580h52m42.91909319s) [noise=true , address=77.90.31.10:4519 , piece_id=AHUZFYFWDH54G4XWM2FW56UTRUCFZ6B2CFSXQEYZ67B2BBTIFY4Q, seg_pos={0 0}, upload=509207]: Uploading piece... Dec 29 20:21:43.048342157 (580h52m42.919425202s) [noise=true , address=145.79.227.14:4840 , piece_id=HLRLJWIF3CV2CKZGXYMLNBO2YTD6DOYDSRRLDSUQKMGXFLU2MIPA, seg_pos={0 0}, upload=509207]: Uploading piece... Dec 29 20:21:43.049040837 (580h52m42.920123882s) [noise=true , address=45.91.116.9:4262 , piece_id=QSGWHDQDGDBSF3GFQYQIQXUNDYGG3MC72QC4R52KBHFZAYXBOK4Q, seg_pos={0 0}, upload=509207]: Uploading piece... Dec 29 20:21:43.049783241 (580h52m42.920866276s) [noise=true , address=145.79.227.34:4906 , piece_id=UWDOJZ6SZ44CU3OMKCDDLHKUXHR6R3N77SZRC3SG4NVXE4G7XTAQ, seg_pos={0 0}, upload=509207]: Uploading piece... Dec 29 20:21:44.733752709 (580h52m44.604835754s) [upload=509207]: Next returned nil segment Dec 29 20:21:44.733773028 (580h52m44.604856063s) [upload=509207]: Waiting for error group managing segments... Dec 29 20:21:45.018632965 (580h52m44.88971601s) [noise=true , address=145.79.227.6:4257 , piece_id=ZUQK7DGTYNMCBOBEL3CB5IS2VPR4GQWHTYK5W23L2MHITQZJMCEQ, seg_pos={0 0}, upload=509207]: Done uploading piece. err: <nil> Dec 29 20:21:45.048385819 (580h52m44.919468854s) [noise=true , address=145.79.227.2:4114 , piece_id=M2Y2S4FAQRCAFPQDQQX2JPCFOD5ODGXKKDCTODIJNWXCFGTG5DBA, seg_pos={0 0}, upload=509207]: Done uploading piece. err: <nil> Dec 29 20:21:45.057250780 (580h52m44.928333815s) [noise=true , address=145.79.227.11:4730 , piece_id=2ER4QY6EOV4V6N5KHFRE6RPC5NGS3UCWR37WSXX2DKLOOLCTI6XQ, seg_pos={0 0}, upload=509207]: Done uploading piece. err: <nil> Dec 29 20:21:45.078675265 (580h52m44.949758311s) [noise=true , address=145.79.227.47:4266 , piece_id=K4RUZUJHMHOPKF4JAX54XFKNG66X6HKHD6FHGABMRDKAUMYGXH5A, seg_pos={0 0}, upload=509207]: Done uploading piece. err: <nil> Dec 29 20:21:45.157138352 (580h52m45.028221397s) [noise=true , address=145.79.227.14:4840 , piece_id=HLRLJWIF3CV2CKZGXYMLNBO2YTD6DOYDSRRLDSUQKMGXFLU2MIPA, seg_pos={0 0}, upload=509207]: Done uploading piece. err: <nil> Dec 29 20:21:45.225140328 (580h52m45.096223363s) [noise=true , address=77.90.31.19:4842 , piece_id=FJC3VHIH6C2CGQRF3GCK2I4IKOEFBZVHEGGY5RWTW5WVPCE27BLA, seg_pos={0 0}, upload=509207]: Done uploading piece. err: <nil> Dec 29 20:21:45.258788668 (580h52m45.129871703s) [noise=true , address=77.90.31.16:4736 , piece_id=QIYITDUU2XGY365LQVWLRAVOQB6HJ7DA4OVGY2NRSHEQRL57VI5A, seg_pos={0 0}, upload=509207]: Done uploading piece. err: <nil> Dec 29 20:21:45.276519862 (580h52m45.147602907s) [noise=true , address=45.91.116.9:4262 , piece_id=QSGWHDQDGDBSF3GFQYQIQXUNDYGG3MC72QC4R52KBHFZAYXBOK4Q, seg_pos={0 0}, upload=509207]: Done uploading piece. err: <nil> Dec 29 20:21:45.286221385 (580h52m45.15730442s) [noise=true , address=77.90.31.10:4519 , piece_id=AHUZFYFWDH54G4XWM2FW56UTRUCFZ6B2CFSXQEYZ67B2BBTIFY4Q, seg_pos={0 0}, upload=509207]: Done uploading piece. err: <nil> Dec 29 20:21:45.321639817 (580h52m45.192722862s) [noise=true , address=77.90.31.18:4808 , piece_id=XZBEJFFOCX6ZDX6FAKT6TZ4N6I44DRV6HU6BHPRFK34CEC4R722Q, seg_pos={0 0}, upload=509207]: Done uploading piece. err: <nil> Dec 29 20:21:45.335734296 (580h52m45.206817331s) [noise=true , address=45.91.116.11:4119 , piece_id=OBVWSCNGH4VME6SU63QSGEPC6T65WC6325B4JR2LHTEN5YPNYQKA, seg_pos={0 0}, upload=509207]: Done uploading piece. err: <nil> Dec 29 20:21:45.338167799 (580h52m45.209250834s) [noise=true , address=77.90.31.13:4629 , piece_id=ZFD3VXFJCRK6VUR2LWE4HR74OH5DSL2BGA2RBYDKRQH2LELSBTEA, seg_pos={0 0}, upload=509207]: Done uploading piece. err: <nil> Dec 29 20:21:45.370591519 (580h52m45.241674564s) [noise=true , address=45.91.116.5:4298 , piece_id=NNYXQBCIK276MMFFNNSF55445ND6ZWNDS2D2FZRI6I7FFN6DYO4A, seg_pos={0 0}, upload=509207]: Done uploading piece. err: <nil> Dec 29 20:21:45.370612739 (580h52m45.241695774s) [seg_pos={0 0}, upload=509207]: Segment reached optimal threshold of 13 pieces. Dec 29 20:21:45.370859700 (580h52m45.241942765s) [noise=true , address=145.79.227.34:4906 , piece_id=UWDOJZ6SZ44CU3OMKCDDLHKUXHR6R3N77SZRC3SG4NVXE4G7XTAQ, seg_pos={0 0}, upload=509207]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw): protocol: expected piece hash; context canceled; EOF Dec 29 20:21:45.370870681 (580h52m45.241953716s) [noise=true , address=145.79.227.34:4906 , piece_id=UWDOJZ6SZ44CU3OMKCDDLHKUXHR6R3N77SZRC3SG4NVXE4G7XTAQ, seg_pos={0 0}, upload=509207]: Long tail cancellation due to optimal threshold Dec 29 20:21:45.370910296 (580h52m45.241993341s) [noise=true , address=145.79.227.49:4337 , piece_id=7MLBIBNLWZNMBHKNPHT4P6TDXDE4NIEPHIPIOTPO3C6ULSX65BMQ, seg_pos={0 0}, upload=509207]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nTZFfzNCcVAjut7NXf9seifysJDfvXkS8aZSnXKfw6gh1TrjE): protocol: expected piece hash; context canceled; EOF Dec 29 20:21:45.370919233 (580h52m45.242002268s) [noise=true , address=145.79.227.49:4337 , piece_id=7MLBIBNLWZNMBHKNPHT4P6TDXDE4NIEPHIPIOTPO3C6ULSX65BMQ, seg_pos={0 0}, upload=509207]: Long tail cancellation due to optimal threshold Dec 29 20:21:45.371157677 (580h52m45.242240722s) [noise=true , address=45.91.116.7:4011 , piece_id=DDIKOM35SDUNGMSHPW2JJFIJ65BU6YWKMYJRXOGYMAHRUYS4GKNA, seg_pos={0 0}, upload=509207]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf): protocol: context canceled; context canceled Dec 29 20:21:45.371166624 (580h52m45.242249659s) [noise=true , address=45.91.116.7:4011 , piece_id=DDIKOM35SDUNGMSHPW2JJFIJ65BU6YWKMYJRXOGYMAHRUYS4GKNA, seg_pos={0 0}, upload=509207]: Long tail cancellation due to optimal threshold Dec 29 20:21:45.371187755 (580h52m45.2422708s) [seg_pos={0 0}, upload=509207]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 20:21:45.371268829 (580h52m45.242351874s) [upload=509207]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 20:21:45.371277245 (580h52m45.24236029s) [upload=509207]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 20:21:45.669010465 (580h52m45.5400935s) [upload=509207]: Done uploading segments... Dec 29 20:21:45.669034200 (580h52m45.540117235s) [upload=509207]: Upload finished. err: <nil> Duration: 30.002891158s core uplink us-select-1
checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing) antiblip: 1 out of 60 is failed (threshold=6) [.......................X....................................] duration: 254.050642ms remote: "tcp:207.211.208.129:443" target: https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt trace: https://jaeger.dev.storj.io/trace/4679501fce62ca7 FAIL (2025-12-29 20:21:54 UTC) orbiter warning: Get "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt": context deadline exceeded Duration: 254.079627ms edge linksharing bgp eu1
checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing) antiblip: 1 out of 60 is failed (threshold=6) [........................X...................................] duration: 257.401882ms remote: "tcp:207.211.208.129:443" target: https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt trace: https://jaeger.dev.storj.io/trace/4679501fce62ca7 FAIL (2025-12-29 20:21:26 UTC) orbiter warning: Get "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt": context deadline exceeded Duration: 257.423243ms edge linksharing bgp eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [X...X.X....X..X.X....X.......X.....................X......X.] duration: 6.020204541s trace: https://jaeger.dev.storj.io/trace/1c0a9205b21ae1d9 FAIL (2025-12-29 20:21:20 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, Piece ID: QT2OC5TP7B4AHQUSET37AJPTK63RYEDCCYQTXJK3PXZLUCNE6CTA) context deadline exceeded; piecestore: (Node ID: 1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, Piece ID: MMDA6PWPI4BDH43ASTZPR7O4QTURU7M2IAMJMWVHFXYRHKWHI6UQ) context deadline exceeded ======LOGS====== Dec 29 20:20:27.131313687 (580h51m27.002396722s) [upload=509184]: Starting upload Dec 29 20:20:27.131389141 (580h51m27.002472186s) [upload=509184]: Done starting upload Dec 29 20:20:27.131559475 (580h51m27.00264251s) [upload=509184]: Uploading segments... Dec 29 20:20:27.131591476 (580h51m27.002674511s) [upload=509184]: Got next segment. Inline: false Dec 29 20:20:27.131616023 (580h51m27.002699068s) [upload=509184]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 20:20:27.131620511 (580h51m27.002703546s) [upload=509184]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 20:20:27.377233387 (580h51m27.248316432s) [seg_pos={0 0}, upload=509184]: Begin upload segment... Dec 29 20:20:27.377293191 (580h51m27.248376256s) [seg_pos={0 0}, upload=509184]: Done begin upload segment. Dec 29 20:20:27.377348086 (580h51m27.248431121s) [noise=true , address=45.91.116.6:4973 , piece_id=OQXF3FRNT3VTTWPDU4VUEETQZ4KOMKYEKWZZI5KMAUA2W2JQUKHQ, seg_pos={0 0}, upload=509184]: Uploading piece... Dec 29 20:20:27.377356201 (580h51m27.248439236s) [noise=true , address=77.90.31.11:4554 , piece_id=QT2OC5TP7B4AHQUSET37AJPTK63RYEDCCYQTXJK3PXZLUCNE6CTA, seg_pos={0 0}, upload=509184]: Uploading piece... Dec 29 20:20:27.377791399 (580h51m27.248874434s) [noise=true , address=45.91.116.11:4119 , piece_id=MXFKHQ4QVQUGXMXVJCPV2XBFQU77Z3UEXUBRF4G4WIRABSFMZ54Q, seg_pos={0 0}, upload=509184]: Uploading piece... Dec 29 20:20:27.378098174 (580h51m27.249181209s) [noise=true , address=45.91.116.5:4297 , piece_id=7LBSEFNFY4Z7HGDYN4ND5INFHI4EQUAK6ZIJ3AD3LM4JECRR337A, seg_pos={0 0}, upload=509184]: Uploading piece... Dec 29 20:20:27.378562498 (580h51m27.249645543s) [noise=true , address=45.91.116.10:4084 , piece_id=SEW5RMRFCJWL6S2EKAATF5IDKC6JZPGJZMQUXY2YNOM2JMHRKQ4Q, seg_pos={0 0}, upload=509184]: Uploading piece... Dec 29 20:20:27.379013447 (580h51m27.250096482s) [noise=true , address=145.79.227.16:4912 , piece_id=HPKV3TAUH4ZPOP6CGQVYWYRFS6IWQCBR2TLQ2MP55M3BRXVMQGMQ, seg_pos={0 0}, upload=509184]: Uploading piece... Dec 29 20:20:27.379426413 (580h51m27.250509468s) [noise=true , address=145.79.227.38:4979 , piece_id=63347TKFC66DXATPZBMBDJI2SRGSVRMFK5PJX6XWKVNZW37RLXXQ, seg_pos={0 0}, upload=509184]: Uploading piece... Dec 29 20:20:27.379870408 (580h51m27.250953443s) [noise=true , address=145.79.227.42:4123 , piece_id=SRIMTLHUON43OL54YIPHX7EXJE6SRXRI6HLQFPJP2OHG4EBJVCXA, seg_pos={0 0}, upload=509184]: Uploading piece... Dec 29 20:20:27.381056066 (580h51m27.252139111s) [noise=true , address=145.79.227.11:4730 , piece_id=3GVO66KJO23B3ZO4SL3FEUNF5B2ZFEXVEHBOYQ5K4Y57UZR6WTMA, seg_pos={0 0}, upload=509184]: Uploading piece... Dec 29 20:20:27.381196664 (580h51m27.252279699s) [noise=true , address=45.91.116.9:4261 , piece_id=MMDA6PWPI4BDH43ASTZPR7O4QTURU7M2IAMJMWVHFXYRHKWHI6UQ, seg_pos={0 0}, upload=509184]: Uploading piece... Dec 29 20:20:27.382535984 (580h51m27.253619029s) [noise=true , address=145.79.227.10:4402 , piece_id=XVFUGPITP3XFL2UOGSID6UR5MCNJD5EWIDSO6ZPOGW5GUQ4JH33Q, seg_pos={0 0}, upload=509184]: Uploading piece... Dec 29 20:20:27.385308283 (580h51m27.256391318s) [noise=true , address=145.79.227.50:4375 , piece_id=QIOWTJ2G3Q3XYIXAFS4TQS76S6JIXFI7AKXVGG5IXD46STFBXVAA, seg_pos={0 0}, upload=509184]: Uploading piece... Dec 29 20:20:27.387233899 (580h51m27.258316954s) [noise=true , address=45.91.116.4:4940 , piece_id=C46WBLZJSL6B3ZON7UTTTKSMBEN66EYDXKQ5RHVWS2RLTFXHIHEA, seg_pos={0 0}, upload=509184]: Uploading piece... Dec 29 20:20:27.388829277 (580h51m27.259912312s) [noise=true , address=145.79.227.36:4519 , piece_id=NJTPKU7HONCDDLEOFZPU555NKYMFNMJR6LKMTAN3WH5REHNFN7PA, seg_pos={0 0}, upload=509184]: Uploading piece... Dec 29 20:20:27.390150192 (580h51m27.261233238s) [noise=true , address=77.90.31.10:4519 , piece_id=DMBKUVAP4ZOKACJIJXLIDCHCVAFTPVMTATGPY7GJZHMZEF53S4JA, seg_pos={0 0}, upload=509184]: Uploading piece... Dec 29 20:20:27.391082859 (580h51m27.262165904s) [noise=true , address=45.91.116.14:4190 , piece_id=ATUQVRD7H4QUESNFUNH2PUFCDXJTRDW3XB26LCRU3KPFD64YMJWQ, seg_pos={0 0}, upload=509184]: Uploading piece... Dec 29 20:20:28.975858587 (580h51m28.846941632s) [upload=509184]: Next returned nil segment Dec 29 20:20:28.975895788 (580h51m28.846978823s) [upload=509184]: Waiting for error group managing segments... Dec 29 20:20:29.357344675 (580h51m29.22842772s) [noise=true , address=145.79.227.42:4123 , piece_id=SRIMTLHUON43OL54YIPHX7EXJE6SRXRI6HLQFPJP2OHG4EBJVCXA, seg_pos={0 0}, upload=509184]: Done uploading piece. err: <nil> Dec 29 20:20:29.371443472 (580h51m29.242526507s) [noise=true , address=145.79.227.50:4375 , piece_id=QIOWTJ2G3Q3XYIXAFS4TQS76S6JIXFI7AKXVGG5IXD46STFBXVAA, seg_pos={0 0}, upload=509184]: Done uploading piece. err: <nil> Dec 29 20:20:29.371538803 (580h51m29.242621838s) [noise=true , address=145.79.227.38:4979 , piece_id=63347TKFC66DXATPZBMBDJI2SRGSVRMFK5PJX6XWKVNZW37RLXXQ, seg_pos={0 0}, upload=509184]: Done uploading piece. err: <nil> Dec 29 20:20:29.373278545 (580h51m29.24436159s) [noise=true , address=145.79.227.11:4730 , piece_id=3GVO66KJO23B3ZO4SL3FEUNF5B2ZFEXVEHBOYQ5K4Y57UZR6WTMA, seg_pos={0 0}, upload=509184]: Done uploading piece. err: <nil> Dec 29 20:20:29.378782747 (580h51m29.249865782s) [noise=true , address=145.79.227.10:4402 , piece_id=XVFUGPITP3XFL2UOGSID6UR5MCNJD5EWIDSO6ZPOGW5GUQ4JH33Q, seg_pos={0 0}, upload=509184]: Done uploading piece. err: <nil> Dec 29 20:20:29.389518659 (580h51m29.260601704s) [noise=true , address=145.79.227.16:4912 , piece_id=HPKV3TAUH4ZPOP6CGQVYWYRFS6IWQCBR2TLQ2MP55M3BRXVMQGMQ, seg_pos={0 0}, upload=509184]: Done uploading piece. err: <nil> Dec 29 20:20:29.405247880 (580h51m29.276330915s) [noise=true , address=145.79.227.36:4519 , piece_id=NJTPKU7HONCDDLEOFZPU555NKYMFNMJR6LKMTAN3WH5REHNFN7PA, seg_pos={0 0}, upload=509184]: Done uploading piece. err: <nil> Dec 29 20:20:29.599390899 (580h51m29.470473934s) [noise=true , address=45.91.116.11:4119 , piece_id=MXFKHQ4QVQUGXMXVJCPV2XBFQU77Z3UEXUBRF4G4WIRABSFMZ54Q, seg_pos={0 0}, upload=509184]: Done uploading piece. err: <nil> Dec 29 20:20:29.611217127 (580h51m29.482300182s) [noise=true , address=45.91.116.9:4261 , piece_id=MMDA6PWPI4BDH43ASTZPR7O4QTURU7M2IAMJMWVHFXYRHKWHI6UQ, seg_pos={0 0}, upload=509184]: Done uploading piece. err: <nil> Dec 29 20:20:29.625312588 (580h51m29.496395633s) [noise=true , address=77.90.31.10:4519 , piece_id=DMBKUVAP4ZOKACJIJXLIDCHCVAFTPVMTATGPY7GJZHMZEF53S4JA, seg_pos={0 0}, upload=509184]: Done uploading piece. err: <nil> Dec 29 20:20:29.627131331 (580h51m29.498214366s) [noise=true , address=45.91.116.5:4297 , piece_id=7LBSEFNFY4Z7HGDYN4ND5INFHI4EQUAK6ZIJ3AD3LM4JECRR337A, seg_pos={0 0}, upload=509184]: Done uploading piece. err: <nil> Dec 29 20:20:29.630897191 (580h51m29.501980236s) [noise=true , address=45.91.116.6:4973 , piece_id=OQXF3FRNT3VTTWPDU4VUEETQZ4KOMKYEKWZZI5KMAUA2W2JQUKHQ, seg_pos={0 0}, upload=509184]: Done uploading piece. err: <nil> Dec 29 20:20:29.661260949 (580h51m29.532343984s) [noise=true , address=77.90.31.11:4554 , piece_id=QT2OC5TP7B4AHQUSET37AJPTK63RYEDCCYQTXJK3PXZLUCNE6CTA, seg_pos={0 0}, upload=509184]: Done uploading piece. err: <nil> Dec 29 20:20:29.661280517 (580h51m29.532363552s) [seg_pos={0 0}, upload=509184]: Segment reached optimal threshold of 13 pieces. Dec 29 20:20:29.661495766 (580h51m29.532578801s) [noise=true , address=45.91.116.14:4190 , piece_id=ATUQVRD7H4QUESNFUNH2PUFCDXJTRDW3XB26LCRU3KPFD64YMJWQ, seg_pos={0 0}, upload=509184]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG): protocol: expected piece hash; context canceled; EOF Dec 29 20:20:29.661506566 (580h51m29.532589602s) [noise=true , address=45.91.116.14:4190 , piece_id=ATUQVRD7H4QUESNFUNH2PUFCDXJTRDW3XB26LCRU3KPFD64YMJWQ, seg_pos={0 0}, upload=509184]: Long tail cancellation due to optimal threshold Dec 29 20:20:29.661644960 (580h51m29.532727995s) [noise=true , address=45.91.116.4:4940 , piece_id=C46WBLZJSL6B3ZON7UTTTKSMBEN66EYDXKQ5RHVWS2RLTFXHIHEA, seg_pos={0 0}, upload=509184]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: expected piece hash; context canceled; EOF Dec 29 20:20:29.661666662 (580h51m29.532749697s) [noise=true , address=45.91.116.4:4940 , piece_id=C46WBLZJSL6B3ZON7UTTTKSMBEN66EYDXKQ5RHVWS2RLTFXHIHEA, seg_pos={0 0}, upload=509184]: Long tail cancellation due to optimal threshold Dec 29 20:20:29.661758857 (580h51m29.532841892s) [noise=true , address=45.91.116.10:4084 , piece_id=SEW5RMRFCJWL6S2EKAATF5IDKC6JZPGJZMQUXY2YNOM2JMHRKQ4Q, seg_pos={0 0}, upload=509184]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj): protocol: expected piece hash; context canceled; EOF Dec 29 20:20:29.661766642 (580h51m29.532849677s) [noise=true , address=45.91.116.10:4084 , piece_id=SEW5RMRFCJWL6S2EKAATF5IDKC6JZPGJZMQUXY2YNOM2JMHRKQ4Q, seg_pos={0 0}, upload=509184]: Long tail cancellation due to optimal threshold Dec 29 20:20:29.661787792 (580h51m29.532870827s) [seg_pos={0 0}, upload=509184]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 20:20:29.661919282 (580h51m29.533002318s) [upload=509184]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 20:20:29.661925063 (580h51m29.533008098s) [upload=509184]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 20:20:29.953169187 (580h51m29.824252222s) [upload=509184]: Done uploading segments... Dec 29 20:20:29.953193053 (580h51m29.824276088s) [upload=509184]: Upload finished. err: <nil> Duration: 6.020232344s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [XX...X.X....X..X.X....X.......X.....................X......X] duration: 30.00302591s trace: https://jaeger.dev.storj.io/trace/1c0a9205b21ae1d9 FAIL (2025-12-29 20:20:56 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, Piece ID: QT2OC5TP7B4AHQUSET37AJPTK63RYEDCCYQTXJK3PXZLUCNE6CTA) context deadline exceeded; piecestore: (Node ID: 1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, Piece ID: MMDA6PWPI4BDH43ASTZPR7O4QTURU7M2IAMJMWVHFXYRHKWHI6UQ) context deadline exceeded ======LOGS====== Dec 29 20:20:27.131313687 (580h51m27.002396722s) [upload=509184]: Starting upload Dec 29 20:20:27.131389141 (580h51m27.002472186s) [upload=509184]: Done starting upload Dec 29 20:20:27.131559475 (580h51m27.00264251s) [upload=509184]: Uploading segments... Dec 29 20:20:27.131591476 (580h51m27.002674511s) [upload=509184]: Got next segment. Inline: false Dec 29 20:20:27.131616023 (580h51m27.002699068s) [upload=509184]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 20:20:27.131620511 (580h51m27.002703546s) [upload=509184]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 20:20:27.377233387 (580h51m27.248316432s) [seg_pos={0 0}, upload=509184]: Begin upload segment... Dec 29 20:20:27.377293191 (580h51m27.248376256s) [seg_pos={0 0}, upload=509184]: Done begin upload segment. Dec 29 20:20:27.377348086 (580h51m27.248431121s) [noise=true , address=45.91.116.6:4973 , piece_id=OQXF3FRNT3VTTWPDU4VUEETQZ4KOMKYEKWZZI5KMAUA2W2JQUKHQ, seg_pos={0 0}, upload=509184]: Uploading piece... Dec 29 20:20:27.377356201 (580h51m27.248439236s) [noise=true , address=77.90.31.11:4554 , piece_id=QT2OC5TP7B4AHQUSET37AJPTK63RYEDCCYQTXJK3PXZLUCNE6CTA, seg_pos={0 0}, upload=509184]: Uploading piece... Dec 29 20:20:27.377791399 (580h51m27.248874434s) [noise=true , address=45.91.116.11:4119 , piece_id=MXFKHQ4QVQUGXMXVJCPV2XBFQU77Z3UEXUBRF4G4WIRABSFMZ54Q, seg_pos={0 0}, upload=509184]: Uploading piece... Dec 29 20:20:27.378098174 (580h51m27.249181209s) [noise=true , address=45.91.116.5:4297 , piece_id=7LBSEFNFY4Z7HGDYN4ND5INFHI4EQUAK6ZIJ3AD3LM4JECRR337A, seg_pos={0 0}, upload=509184]: Uploading piece... Dec 29 20:20:27.378562498 (580h51m27.249645543s) [noise=true , address=45.91.116.10:4084 , piece_id=SEW5RMRFCJWL6S2EKAATF5IDKC6JZPGJZMQUXY2YNOM2JMHRKQ4Q, seg_pos={0 0}, upload=509184]: Uploading piece... Dec 29 20:20:27.379013447 (580h51m27.250096482s) [noise=true , address=145.79.227.16:4912 , piece_id=HPKV3TAUH4ZPOP6CGQVYWYRFS6IWQCBR2TLQ2MP55M3BRXVMQGMQ, seg_pos={0 0}, upload=509184]: Uploading piece... Dec 29 20:20:27.379426413 (580h51m27.250509468s) [noise=true , address=145.79.227.38:4979 , piece_id=63347TKFC66DXATPZBMBDJI2SRGSVRMFK5PJX6XWKVNZW37RLXXQ, seg_pos={0 0}, upload=509184]: Uploading piece... Dec 29 20:20:27.379870408 (580h51m27.250953443s) [noise=true , address=145.79.227.42:4123 , piece_id=SRIMTLHUON43OL54YIPHX7EXJE6SRXRI6HLQFPJP2OHG4EBJVCXA, seg_pos={0 0}, upload=509184]: Uploading piece... Dec 29 20:20:27.381056066 (580h51m27.252139111s) [noise=true , address=145.79.227.11:4730 , piece_id=3GVO66KJO23B3ZO4SL3FEUNF5B2ZFEXVEHBOYQ5K4Y57UZR6WTMA, seg_pos={0 0}, upload=509184]: Uploading piece... Dec 29 20:20:27.381196664 (580h51m27.252279699s) [noise=true , address=45.91.116.9:4261 , piece_id=MMDA6PWPI4BDH43ASTZPR7O4QTURU7M2IAMJMWVHFXYRHKWHI6UQ, seg_pos={0 0}, upload=509184]: Uploading piece... Dec 29 20:20:27.382535984 (580h51m27.253619029s) [noise=true , address=145.79.227.10:4402 , piece_id=XVFUGPITP3XFL2UOGSID6UR5MCNJD5EWIDSO6ZPOGW5GUQ4JH33Q, seg_pos={0 0}, upload=509184]: Uploading piece... Dec 29 20:20:27.385308283 (580h51m27.256391318s) [noise=true , address=145.79.227.50:4375 , piece_id=QIOWTJ2G3Q3XYIXAFS4TQS76S6JIXFI7AKXVGG5IXD46STFBXVAA, seg_pos={0 0}, upload=509184]: Uploading piece... Dec 29 20:20:27.387233899 (580h51m27.258316954s) [noise=true , address=45.91.116.4:4940 , piece_id=C46WBLZJSL6B3ZON7UTTTKSMBEN66EYDXKQ5RHVWS2RLTFXHIHEA, seg_pos={0 0}, upload=509184]: Uploading piece... Dec 29 20:20:27.388829277 (580h51m27.259912312s) [noise=true , address=145.79.227.36:4519 , piece_id=NJTPKU7HONCDDLEOFZPU555NKYMFNMJR6LKMTAN3WH5REHNFN7PA, seg_pos={0 0}, upload=509184]: Uploading piece... Dec 29 20:20:27.390150192 (580h51m27.261233238s) [noise=true , address=77.90.31.10:4519 , piece_id=DMBKUVAP4ZOKACJIJXLIDCHCVAFTPVMTATGPY7GJZHMZEF53S4JA, seg_pos={0 0}, upload=509184]: Uploading piece... Dec 29 20:20:27.391082859 (580h51m27.262165904s) [noise=true , address=45.91.116.14:4190 , piece_id=ATUQVRD7H4QUESNFUNH2PUFCDXJTRDW3XB26LCRU3KPFD64YMJWQ, seg_pos={0 0}, upload=509184]: Uploading piece... Dec 29 20:20:28.975858587 (580h51m28.846941632s) [upload=509184]: Next returned nil segment Dec 29 20:20:28.975895788 (580h51m28.846978823s) [upload=509184]: Waiting for error group managing segments... Dec 29 20:20:29.357344675 (580h51m29.22842772s) [noise=true , address=145.79.227.42:4123 , piece_id=SRIMTLHUON43OL54YIPHX7EXJE6SRXRI6HLQFPJP2OHG4EBJVCXA, seg_pos={0 0}, upload=509184]: Done uploading piece. err: <nil> Dec 29 20:20:29.371443472 (580h51m29.242526507s) [noise=true , address=145.79.227.50:4375 , piece_id=QIOWTJ2G3Q3XYIXAFS4TQS76S6JIXFI7AKXVGG5IXD46STFBXVAA, seg_pos={0 0}, upload=509184]: Done uploading piece. err: <nil> Dec 29 20:20:29.371538803 (580h51m29.242621838s) [noise=true , address=145.79.227.38:4979 , piece_id=63347TKFC66DXATPZBMBDJI2SRGSVRMFK5PJX6XWKVNZW37RLXXQ, seg_pos={0 0}, upload=509184]: Done uploading piece. err: <nil> Dec 29 20:20:29.373278545 (580h51m29.24436159s) [noise=true , address=145.79.227.11:4730 , piece_id=3GVO66KJO23B3ZO4SL3FEUNF5B2ZFEXVEHBOYQ5K4Y57UZR6WTMA, seg_pos={0 0}, upload=509184]: Done uploading piece. err: <nil> Dec 29 20:20:29.378782747 (580h51m29.249865782s) [noise=true , address=145.79.227.10:4402 , piece_id=XVFUGPITP3XFL2UOGSID6UR5MCNJD5EWIDSO6ZPOGW5GUQ4JH33Q, seg_pos={0 0}, upload=509184]: Done uploading piece. err: <nil> Dec 29 20:20:29.389518659 (580h51m29.260601704s) [noise=true , address=145.79.227.16:4912 , piece_id=HPKV3TAUH4ZPOP6CGQVYWYRFS6IWQCBR2TLQ2MP55M3BRXVMQGMQ, seg_pos={0 0}, upload=509184]: Done uploading piece. err: <nil> Dec 29 20:20:29.405247880 (580h51m29.276330915s) [noise=true , address=145.79.227.36:4519 , piece_id=NJTPKU7HONCDDLEOFZPU555NKYMFNMJR6LKMTAN3WH5REHNFN7PA, seg_pos={0 0}, upload=509184]: Done uploading piece. err: <nil> Dec 29 20:20:29.599390899 (580h51m29.470473934s) [noise=true , address=45.91.116.11:4119 , piece_id=MXFKHQ4QVQUGXMXVJCPV2XBFQU77Z3UEXUBRF4G4WIRABSFMZ54Q, seg_pos={0 0}, upload=509184]: Done uploading piece. err: <nil> Dec 29 20:20:29.611217127 (580h51m29.482300182s) [noise=true , address=45.91.116.9:4261 , piece_id=MMDA6PWPI4BDH43ASTZPR7O4QTURU7M2IAMJMWVHFXYRHKWHI6UQ, seg_pos={0 0}, upload=509184]: Done uploading piece. err: <nil> Dec 29 20:20:29.625312588 (580h51m29.496395633s) [noise=true , address=77.90.31.10:4519 , piece_id=DMBKUVAP4ZOKACJIJXLIDCHCVAFTPVMTATGPY7GJZHMZEF53S4JA, seg_pos={0 0}, upload=509184]: Done uploading piece. err: <nil> Dec 29 20:20:29.627131331 (580h51m29.498214366s) [noise=true , address=45.91.116.5:4297 , piece_id=7LBSEFNFY4Z7HGDYN4ND5INFHI4EQUAK6ZIJ3AD3LM4JECRR337A, seg_pos={0 0}, upload=509184]: Done uploading piece. err: <nil> Dec 29 20:20:29.630897191 (580h51m29.501980236s) [noise=true , address=45.91.116.6:4973 , piece_id=OQXF3FRNT3VTTWPDU4VUEETQZ4KOMKYEKWZZI5KMAUA2W2JQUKHQ, seg_pos={0 0}, upload=509184]: Done uploading piece. err: <nil> Dec 29 20:20:29.661260949 (580h51m29.532343984s) [noise=true , address=77.90.31.11:4554 , piece_id=QT2OC5TP7B4AHQUSET37AJPTK63RYEDCCYQTXJK3PXZLUCNE6CTA, seg_pos={0 0}, upload=509184]: Done uploading piece. err: <nil> Dec 29 20:20:29.661280517 (580h51m29.532363552s) [seg_pos={0 0}, upload=509184]: Segment reached optimal threshold of 13 pieces. Dec 29 20:20:29.661495766 (580h51m29.532578801s) [noise=true , address=45.91.116.14:4190 , piece_id=ATUQVRD7H4QUESNFUNH2PUFCDXJTRDW3XB26LCRU3KPFD64YMJWQ, seg_pos={0 0}, upload=509184]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG): protocol: expected piece hash; context canceled; EOF Dec 29 20:20:29.661506566 (580h51m29.532589602s) [noise=true , address=45.91.116.14:4190 , piece_id=ATUQVRD7H4QUESNFUNH2PUFCDXJTRDW3XB26LCRU3KPFD64YMJWQ, seg_pos={0 0}, upload=509184]: Long tail cancellation due to optimal threshold Dec 29 20:20:29.661644960 (580h51m29.532727995s) [noise=true , address=45.91.116.4:4940 , piece_id=C46WBLZJSL6B3ZON7UTTTKSMBEN66EYDXKQ5RHVWS2RLTFXHIHEA, seg_pos={0 0}, upload=509184]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: expected piece hash; context canceled; EOF Dec 29 20:20:29.661666662 (580h51m29.532749697s) [noise=true , address=45.91.116.4:4940 , piece_id=C46WBLZJSL6B3ZON7UTTTKSMBEN66EYDXKQ5RHVWS2RLTFXHIHEA, seg_pos={0 0}, upload=509184]: Long tail cancellation due to optimal threshold Dec 29 20:20:29.661758857 (580h51m29.532841892s) [noise=true , address=45.91.116.10:4084 , piece_id=SEW5RMRFCJWL6S2EKAATF5IDKC6JZPGJZMQUXY2YNOM2JMHRKQ4Q, seg_pos={0 0}, upload=509184]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj): protocol: expected piece hash; context canceled; EOF Dec 29 20:20:29.661766642 (580h51m29.532849677s) [noise=true , address=45.91.116.10:4084 , piece_id=SEW5RMRFCJWL6S2EKAATF5IDKC6JZPGJZMQUXY2YNOM2JMHRKQ4Q, seg_pos={0 0}, upload=509184]: Long tail cancellation due to optimal threshold Dec 29 20:20:29.661787792 (580h51m29.532870827s) [seg_pos={0 0}, upload=509184]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 20:20:29.661919282 (580h51m29.533002318s) [upload=509184]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 20:20:29.661925063 (580h51m29.533008098s) [upload=509184]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 20:20:29.953169187 (580h51m29.824252222s) [upload=509184]: Done uploading segments... Dec 29 20:20:29.953193053 (580h51m29.824276088s) [upload=509184]: Upload finished. err: <nil> Duration: 30.0030415s core uplink us-select-1
checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing) antiblip: 1 out of 60 is failed (threshold=6) [.........................X..................................] duration: 255.953565ms remote: "tcp:207.211.208.129:443" target: https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt trace: https://jaeger.dev.storj.io/trace/4679501fce62ca7 FAIL (2025-12-29 20:20:42 UTC) orbiter warning: Get "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt": context deadline exceeded Duration: 255.992349ms edge linksharing bgp eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [.XX...X.X....X..X.X....X.......X.....................X......] duration: 6.136763796s trace: https://jaeger.dev.storj.io/trace/607cf963331dd581 FAIL (2025-12-29 20:20:09 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 20:14:59.343887631 (580h45m59.214970666s) [upload=509109]: Done starting upload Dec 29 20:14:59.343929450 (580h45m59.215012485s) [upload=509109]: Uploading segments... Dec 29 20:14:59.344178725 (580h45m59.21526176s) [upload=509109]: Got next segment. Inline: false Dec 29 20:14:59.344213341 (580h45m59.215296376s) [upload=509109]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 20:14:59.344218040 (580h45m59.215301075s) [upload=509109]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 20:14:59.620876250 (580h45m59.491959285s) [seg_pos={0 0}, upload=509109]: Begin upload segment... Dec 29 20:14:59.620932867 (580h45m59.492015912s) [seg_pos={0 0}, upload=509109]: Done begin upload segment. Dec 29 20:14:59.620975569 (580h45m59.492058604s) [noise=true , address=145.79.227.51:4411 , piece_id=LYCY2Z7Z3BFR5XSM6JXFCL5LZAU56PUBVGHLVURZUTQTEZF7CLFQ, seg_pos={0 0}, upload=509109]: Uploading piece... Dec 29 20:14:59.620983133 (580h45m59.492066178s) [noise=true , address=145.79.227.48:4303 , piece_id=UYEJZYH7QYLL6KGKEQBH6DEGEBMWZSM2OT7W4V6D4D3FQV4XUDZQ, seg_pos={0 0}, upload=509109]: Uploading piece... Dec 29 20:14:59.621128750 (580h45m59.492211796s) [noise=true , address=145.79.227.46:4602 , piece_id=ZJA75URLP6YW2ROXGLR4PONX2PXXWZNGGANAI37SREOPJ5S7OV3A, seg_pos={0 0}, upload=509109]: Uploading piece... Dec 29 20:14:59.621771063 (580h45m59.492854098s) [noise=true , address=145.79.227.15:4875 , piece_id=RFIPM3MTYFN5PLV4NVNF7AE5OVA54ZIOR2PUY7S4674CN466ECBA, seg_pos={0 0}, upload=509109]: Uploading piece... Dec 29 20:14:59.622036208 (580h45m59.493119263s) [noise=true , address=145.79.227.18:4006 , piece_id=QP3MYNJN7EGPAM7NB4E2ITEV6EPATVAIWGAPT3KNEUXEPPBNQVDA, seg_pos={0 0}, upload=509109]: Uploading piece... Dec 29 20:14:59.622303207 (580h45m59.493386252s) [noise=true , address=145.79.227.28:4725 , piece_id=U442UUBRULRA3AY6H7EP5G2XJFY47YYDMBWRCMQBM2BLCKZJBYDA, seg_pos={0 0}, upload=509109]: Uploading piece... Dec 29 20:14:59.622693610 (580h45m59.493776645s) [noise=true , address=145.79.227.44:4194 , piece_id=NEQVB6ERVDONQCXWRFXXFPZ3EBSLRJJMWOZHSBCKZWFWZ5FJXTNA, seg_pos={0 0}, upload=509109]: Uploading piece... Dec 29 20:14:59.623383705 (580h45m59.49446674s) [noise=true , address=77.90.31.13:4627 , piece_id=OVJXRA53CFJFZSRJ6H2MHZLO4RCLLC3TIGGYQN3ESNQFA3ESQD3A, seg_pos={0 0}, upload=509109]: Uploading piece... Dec 29 20:14:59.623912161 (580h45m59.494995206s) [noise=true , address=45.91.116.7:4010 , piece_id=BO772OMWJBZDCBAX3O6ECC766FEHZVTOJHKJHDLCSXJWKEKIV6BA, seg_pos={0 0}, upload=509109]: Uploading piece... Dec 29 20:14:59.624389991 (580h45m59.495473026s) [noise=true , address=145.79.227.2:4115 , piece_id=J6ZAJU7FYPAFEI47EJBUUUCASWZY2PZZWABSA7WRLAQPYQMFTENQ, seg_pos={0 0}, upload=509109]: Uploading piece... Dec 29 20:14:59.624497966 (580h45m59.495581001s) [noise=true , address=145.79.227.11:4731 , piece_id=46XM74D3UEXNZXHZYWFOYW6SHHWTQU2HA6Q2WYSJLH7OVC23JALQ, seg_pos={0 0}, upload=509109]: Uploading piece... Dec 29 20:14:59.624935829 (580h45m59.496018864s) [noise=true , address=145.79.227.19:4078 , piece_id=NZLVPFPYJXZHMK2XSIMIDBR7CN4OWVWT5QGNLTK3STC5KBY7CMGA, seg_pos={0 0}, upload=509109]: Uploading piece... Dec 29 20:14:59.624977129 (580h45m59.496060174s) [noise=true , address=45.91.116.9:4262 , piece_id=UV5XMT447DE6QTJYRYBD7466OPDQ3YQSIVQ4C2BSIO37OCZKGSHA, seg_pos={0 0}, upload=509109]: Uploading piece... Dec 29 20:14:59.625338717 (580h45m59.496421762s) [noise=true , address=45.91.116.6:4975 , piece_id=YHNIUJWPFG4EC7FJYL43CCVFTKZ2GMKX5ANKEIHDMLAL2AGA7V3A, seg_pos={0 0}, upload=509109]: Uploading piece... Dec 29 20:14:59.625486658 (580h45m59.496569704s) [noise=true , address=145.79.227.36:4518 , piece_id=3DYS7R2VTEUGCDWBWFP2OZHYTMQGLK7OS3OFSCUBIKWSWHJ44ELQ, seg_pos={0 0}, upload=509109]: Uploading piece... Dec 29 20:14:59.625852676 (580h45m59.496935721s) [noise=true , address=145.79.227.9:4366 , piece_id=ICYZM4QQZQJLS4T4KYM47NEMIKYD7PZWQ2U2NEFLAWLLPO6OFG3A, seg_pos={0 0}, upload=509109]: Uploading piece... Dec 29 20:15:01.223830613 (580h46m1.094913658s) [upload=509109]: Next returned nil segment Dec 29 20:15:01.223865600 (580h46m1.094948645s) [upload=509109]: Waiting for error group managing segments... Dec 29 20:15:01.599999537 (580h46m1.471082582s) [noise=true , address=145.79.227.11:4731 , piece_id=46XM74D3UEXNZXHZYWFOYW6SHHWTQU2HA6Q2WYSJLH7OVC23JALQ, seg_pos={0 0}, upload=509109]: Done uploading piece. err: <nil> Dec 29 20:15:01.606734191 (580h46m1.477817236s) [noise=true , address=145.79.227.9:4366 , piece_id=ICYZM4QQZQJLS4T4KYM47NEMIKYD7PZWQ2U2NEFLAWLLPO6OFG3A, seg_pos={0 0}, upload=509109]: Done uploading piece. err: <nil> Dec 29 20:15:01.609790219 (580h46m1.480873265s) [noise=true , address=145.79.227.48:4303 , piece_id=UYEJZYH7QYLL6KGKEQBH6DEGEBMWZSM2OT7W4V6D4D3FQV4XUDZQ, seg_pos={0 0}, upload=509109]: Done uploading piece. err: <nil> Dec 29 20:15:01.613856723 (580h46m1.484939768s) [noise=true , address=145.79.227.51:4411 , piece_id=LYCY2Z7Z3BFR5XSM6JXFCL5LZAU56PUBVGHLVURZUTQTEZF7CLFQ, seg_pos={0 0}, upload=509109]: Done uploading piece. err: <nil> Dec 29 20:15:01.615434938 (580h46m1.486517973s) [noise=true , address=145.79.227.46:4602 , piece_id=ZJA75URLP6YW2ROXGLR4PONX2PXXWZNGGANAI37SREOPJ5S7OV3A, seg_pos={0 0}, upload=509109]: Done uploading piece. err: <nil> Dec 29 20:15:01.617581145 (580h46m1.48866419s) [noise=true , address=145.79.227.19:4078 , piece_id=NZLVPFPYJXZHMK2XSIMIDBR7CN4OWVWT5QGNLTK3STC5KBY7CMGA, seg_pos={0 0}, upload=509109]: Done uploading piece. err: <nil> Dec 29 20:15:01.618039818 (580h46m1.489122863s) [noise=true , address=145.79.227.15:4875 , piece_id=RFIPM3MTYFN5PLV4NVNF7AE5OVA54ZIOR2PUY7S4674CN466ECBA, seg_pos={0 0}, upload=509109]: Done uploading piece. err: <nil> Dec 29 20:15:01.624408906 (580h46m1.495491951s) [noise=true , address=145.79.227.36:4518 , piece_id=3DYS7R2VTEUGCDWBWFP2OZHYTMQGLK7OS3OFSCUBIKWSWHJ44ELQ, seg_pos={0 0}, upload=509109]: Done uploading piece. err: <nil> Dec 29 20:15:01.629708417 (580h46m1.500791482s) [noise=true , address=145.79.227.2:4115 , piece_id=J6ZAJU7FYPAFEI47EJBUUUCASWZY2PZZWABSA7WRLAQPYQMFTENQ, seg_pos={0 0}, upload=509109]: Done uploading piece. err: <nil> Dec 29 20:15:01.654622637 (580h46m1.525705682s) [noise=true , address=145.79.227.18:4006 , piece_id=QP3MYNJN7EGPAM7NB4E2ITEV6EPATVAIWGAPT3KNEUXEPPBNQVDA, seg_pos={0 0}, upload=509109]: Done uploading piece. err: <nil> Dec 29 20:15:01.799321647 (580h46m1.670404682s) [noise=true , address=77.90.31.13:4627 , piece_id=OVJXRA53CFJFZSRJ6H2MHZLO4RCLLC3TIGGYQN3ESNQFA3ESQD3A, seg_pos={0 0}, upload=509109]: Done uploading piece. err: <nil> Dec 29 20:15:01.902753888 (580h46m1.773836943s) [noise=true , address=45.91.116.7:4010 , piece_id=BO772OMWJBZDCBAX3O6ECC766FEHZVTOJHKJHDLCSXJWKEKIV6BA, seg_pos={0 0}, upload=509109]: Done uploading piece. err: <nil> Dec 29 20:15:02.167078173 (580h46m2.038161228s) [noise=true , address=45.91.116.9:4262 , piece_id=UV5XMT447DE6QTJYRYBD7466OPDQ3YQSIVQ4C2BSIO37OCZKGSHA, seg_pos={0 0}, upload=509109]: Done uploading piece. err: <nil> Dec 29 20:15:02.167162874 (580h46m2.038245929s) [seg_pos={0 0}, upload=509109]: Segment reached optimal threshold of 13 pieces. Dec 29 20:15:02.170200207 (580h46m2.041283262s) [noise=true , address=45.91.116.6:4975 , piece_id=YHNIUJWPFG4EC7FJYL43CCVFTKZ2GMKX5ANKEIHDMLAL2AGA7V3A, seg_pos={0 0}, upload=509109]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: read tcp 188.166.225.170:42964->45.91.116.6:4975: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 29 20:15:02.170215235 (580h46m2.04129829s) [noise=true , address=45.91.116.6:4975 , piece_id=YHNIUJWPFG4EC7FJYL43CCVFTKZ2GMKX5ANKEIHDMLAL2AGA7V3A, seg_pos={0 0}, upload=509109]: Long tail cancellation due to optimal threshold Dec 29 20:15:02.170672046 (580h46m2.041755101s) [noise=true , address=145.79.227.44:4194 , piece_id=NEQVB6ERVDONQCXWRFXXFPZ3EBSLRJJMWOZHSBCKZWFWZ5FJXTNA, seg_pos={0 0}, upload=509109]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh): protocol: expected piece hash; context canceled; EOF Dec 29 20:15:02.170683507 (580h46m2.041766553s) [noise=true , address=145.79.227.44:4194 , piece_id=NEQVB6ERVDONQCXWRFXXFPZ3EBSLRJJMWOZHSBCKZWFWZ5FJXTNA, seg_pos={0 0}, upload=509109]: Long tail cancellation due to optimal threshold Dec 29 20:15:02.170846298 (580h46m2.041929353s) [noise=true , address=145.79.227.28:4725 , piece_id=U442UUBRULRA3AY6H7EP5G2XJFY47YYDMBWRCMQBM2BLCKZJBYDA, seg_pos={0 0}, upload=509109]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12YKLTah3swX14kL82eNogJxd8D8Hjfq7SPjDDQAapDnXmDctP1): protocol: write tcp 188.166.225.170:56476->145.79.227.28:4725: use of closed network connection; write tcp 188.166.225.170:56476->145.79.227.28:4725: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56476->145.79.227.28:4725: use of closed network connection Dec 29 20:15:02.170859112 (580h46m2.041942157s) [noise=true , address=145.79.227.28:4725 , piece_id=U442UUBRULRA3AY6H7EP5G2XJFY47YYDMBWRCMQBM2BLCKZJBYDA, seg_pos={0 0}, upload=509109]: Long tail cancellation due to optimal threshold Dec 29 20:15:02.170884811 (580h46m2.041967856s) [seg_pos={0 0}, upload=509109]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 20:15:02.171053743 (580h46m2.042136788s) [upload=509109]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 20:15:02.171062129 (580h46m2.042145184s) [upload=509109]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 20:15:02.455142652 (580h46m2.326225707s) [upload=509109]: Done uploading segments... Dec 29 20:15:02.455171597 (580h46m2.326254642s) [upload=509109]: Upload finished. err: <nil> Duration: 6.136793504s core uplink us-select-1
checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing) antiblip: 1 out of 60 is failed (threshold=6) [..........................X.................................] duration: 288.40088ms remote: "tcp:207.211.208.129:443" target: https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt trace: https://jaeger.dev.storj.io/trace/4679501fce62ca7 FAIL (2025-12-29 20:20:06 UTC) orbiter warning: Get "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt": context deadline exceeded Duration: 288.433692ms edge linksharing bgp eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [X.XX...X.X....X..X.X....X.......X.....................X.....] duration: 28.737138743s trace: https://jaeger.dev.storj.io/trace/607cf963331dd581 FAIL (2025-12-29 20:19:45 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 20:14:59.343887631 (580h45m59.214970666s) [upload=509109]: Done starting upload Dec 29 20:14:59.343929450 (580h45m59.215012485s) [upload=509109]: Uploading segments... Dec 29 20:14:59.344178725 (580h45m59.21526176s) [upload=509109]: Got next segment. Inline: false Dec 29 20:14:59.344213341 (580h45m59.215296376s) [upload=509109]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 20:14:59.344218040 (580h45m59.215301075s) [upload=509109]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 20:14:59.620876250 (580h45m59.491959285s) [seg_pos={0 0}, upload=509109]: Begin upload segment... Dec 29 20:14:59.620932867 (580h45m59.492015912s) [seg_pos={0 0}, upload=509109]: Done begin upload segment. Dec 29 20:14:59.620975569 (580h45m59.492058604s) [noise=true , address=145.79.227.51:4411 , piece_id=LYCY2Z7Z3BFR5XSM6JXFCL5LZAU56PUBVGHLVURZUTQTEZF7CLFQ, seg_pos={0 0}, upload=509109]: Uploading piece... Dec 29 20:14:59.620983133 (580h45m59.492066178s) [noise=true , address=145.79.227.48:4303 , piece_id=UYEJZYH7QYLL6KGKEQBH6DEGEBMWZSM2OT7W4V6D4D3FQV4XUDZQ, seg_pos={0 0}, upload=509109]: Uploading piece... Dec 29 20:14:59.621128750 (580h45m59.492211796s) [noise=true , address=145.79.227.46:4602 , piece_id=ZJA75URLP6YW2ROXGLR4PONX2PXXWZNGGANAI37SREOPJ5S7OV3A, seg_pos={0 0}, upload=509109]: Uploading piece... Dec 29 20:14:59.621771063 (580h45m59.492854098s) [noise=true , address=145.79.227.15:4875 , piece_id=RFIPM3MTYFN5PLV4NVNF7AE5OVA54ZIOR2PUY7S4674CN466ECBA, seg_pos={0 0}, upload=509109]: Uploading piece... Dec 29 20:14:59.622036208 (580h45m59.493119263s) [noise=true , address=145.79.227.18:4006 , piece_id=QP3MYNJN7EGPAM7NB4E2ITEV6EPATVAIWGAPT3KNEUXEPPBNQVDA, seg_pos={0 0}, upload=509109]: Uploading piece... Dec 29 20:14:59.622303207 (580h45m59.493386252s) [noise=true , address=145.79.227.28:4725 , piece_id=U442UUBRULRA3AY6H7EP5G2XJFY47YYDMBWRCMQBM2BLCKZJBYDA, seg_pos={0 0}, upload=509109]: Uploading piece... Dec 29 20:14:59.622693610 (580h45m59.493776645s) [noise=true , address=145.79.227.44:4194 , piece_id=NEQVB6ERVDONQCXWRFXXFPZ3EBSLRJJMWOZHSBCKZWFWZ5FJXTNA, seg_pos={0 0}, upload=509109]: Uploading piece... Dec 29 20:14:59.623383705 (580h45m59.49446674s) [noise=true , address=77.90.31.13:4627 , piece_id=OVJXRA53CFJFZSRJ6H2MHZLO4RCLLC3TIGGYQN3ESNQFA3ESQD3A, seg_pos={0 0}, upload=509109]: Uploading piece... Dec 29 20:14:59.623912161 (580h45m59.494995206s) [noise=true , address=45.91.116.7:4010 , piece_id=BO772OMWJBZDCBAX3O6ECC766FEHZVTOJHKJHDLCSXJWKEKIV6BA, seg_pos={0 0}, upload=509109]: Uploading piece... Dec 29 20:14:59.624389991 (580h45m59.495473026s) [noise=true , address=145.79.227.2:4115 , piece_id=J6ZAJU7FYPAFEI47EJBUUUCASWZY2PZZWABSA7WRLAQPYQMFTENQ, seg_pos={0 0}, upload=509109]: Uploading piece... Dec 29 20:14:59.624497966 (580h45m59.495581001s) [noise=true , address=145.79.227.11:4731 , piece_id=46XM74D3UEXNZXHZYWFOYW6SHHWTQU2HA6Q2WYSJLH7OVC23JALQ, seg_pos={0 0}, upload=509109]: Uploading piece... Dec 29 20:14:59.624935829 (580h45m59.496018864s) [noise=true , address=145.79.227.19:4078 , piece_id=NZLVPFPYJXZHMK2XSIMIDBR7CN4OWVWT5QGNLTK3STC5KBY7CMGA, seg_pos={0 0}, upload=509109]: Uploading piece... Dec 29 20:14:59.624977129 (580h45m59.496060174s) [noise=true , address=45.91.116.9:4262 , piece_id=UV5XMT447DE6QTJYRYBD7466OPDQ3YQSIVQ4C2BSIO37OCZKGSHA, seg_pos={0 0}, upload=509109]: Uploading piece... Dec 29 20:14:59.625338717 (580h45m59.496421762s) [noise=true , address=45.91.116.6:4975 , piece_id=YHNIUJWPFG4EC7FJYL43CCVFTKZ2GMKX5ANKEIHDMLAL2AGA7V3A, seg_pos={0 0}, upload=509109]: Uploading piece... Dec 29 20:14:59.625486658 (580h45m59.496569704s) [noise=true , address=145.79.227.36:4518 , piece_id=3DYS7R2VTEUGCDWBWFP2OZHYTMQGLK7OS3OFSCUBIKWSWHJ44ELQ, seg_pos={0 0}, upload=509109]: Uploading piece... Dec 29 20:14:59.625852676 (580h45m59.496935721s) [noise=true , address=145.79.227.9:4366 , piece_id=ICYZM4QQZQJLS4T4KYM47NEMIKYD7PZWQ2U2NEFLAWLLPO6OFG3A, seg_pos={0 0}, upload=509109]: Uploading piece... Dec 29 20:15:01.223830613 (580h46m1.094913658s) [upload=509109]: Next returned nil segment Dec 29 20:15:01.223865600 (580h46m1.094948645s) [upload=509109]: Waiting for error group managing segments... Dec 29 20:15:01.599999537 (580h46m1.471082582s) [noise=true , address=145.79.227.11:4731 , piece_id=46XM74D3UEXNZXHZYWFOYW6SHHWTQU2HA6Q2WYSJLH7OVC23JALQ, seg_pos={0 0}, upload=509109]: Done uploading piece. err: <nil> Dec 29 20:15:01.606734191 (580h46m1.477817236s) [noise=true , address=145.79.227.9:4366 , piece_id=ICYZM4QQZQJLS4T4KYM47NEMIKYD7PZWQ2U2NEFLAWLLPO6OFG3A, seg_pos={0 0}, upload=509109]: Done uploading piece. err: <nil> Dec 29 20:15:01.609790219 (580h46m1.480873265s) [noise=true , address=145.79.227.48:4303 , piece_id=UYEJZYH7QYLL6KGKEQBH6DEGEBMWZSM2OT7W4V6D4D3FQV4XUDZQ, seg_pos={0 0}, upload=509109]: Done uploading piece. err: <nil> Dec 29 20:15:01.613856723 (580h46m1.484939768s) [noise=true , address=145.79.227.51:4411 , piece_id=LYCY2Z7Z3BFR5XSM6JXFCL5LZAU56PUBVGHLVURZUTQTEZF7CLFQ, seg_pos={0 0}, upload=509109]: Done uploading piece. err: <nil> Dec 29 20:15:01.615434938 (580h46m1.486517973s) [noise=true , address=145.79.227.46:4602 , piece_id=ZJA75URLP6YW2ROXGLR4PONX2PXXWZNGGANAI37SREOPJ5S7OV3A, seg_pos={0 0}, upload=509109]: Done uploading piece. err: <nil> Dec 29 20:15:01.617581145 (580h46m1.48866419s) [noise=true , address=145.79.227.19:4078 , piece_id=NZLVPFPYJXZHMK2XSIMIDBR7CN4OWVWT5QGNLTK3STC5KBY7CMGA, seg_pos={0 0}, upload=509109]: Done uploading piece. err: <nil> Dec 29 20:15:01.618039818 (580h46m1.489122863s) [noise=true , address=145.79.227.15:4875 , piece_id=RFIPM3MTYFN5PLV4NVNF7AE5OVA54ZIOR2PUY7S4674CN466ECBA, seg_pos={0 0}, upload=509109]: Done uploading piece. err: <nil> Dec 29 20:15:01.624408906 (580h46m1.495491951s) [noise=true , address=145.79.227.36:4518 , piece_id=3DYS7R2VTEUGCDWBWFP2OZHYTMQGLK7OS3OFSCUBIKWSWHJ44ELQ, seg_pos={0 0}, upload=509109]: Done uploading piece. err: <nil> Dec 29 20:15:01.629708417 (580h46m1.500791482s) [noise=true , address=145.79.227.2:4115 , piece_id=J6ZAJU7FYPAFEI47EJBUUUCASWZY2PZZWABSA7WRLAQPYQMFTENQ, seg_pos={0 0}, upload=509109]: Done uploading piece. err: <nil> Dec 29 20:15:01.654622637 (580h46m1.525705682s) [noise=true , address=145.79.227.18:4006 , piece_id=QP3MYNJN7EGPAM7NB4E2ITEV6EPATVAIWGAPT3KNEUXEPPBNQVDA, seg_pos={0 0}, upload=509109]: Done uploading piece. err: <nil> Dec 29 20:15:01.799321647 (580h46m1.670404682s) [noise=true , address=77.90.31.13:4627 , piece_id=OVJXRA53CFJFZSRJ6H2MHZLO4RCLLC3TIGGYQN3ESNQFA3ESQD3A, seg_pos={0 0}, upload=509109]: Done uploading piece. err: <nil> Dec 29 20:15:01.902753888 (580h46m1.773836943s) [noise=true , address=45.91.116.7:4010 , piece_id=BO772OMWJBZDCBAX3O6ECC766FEHZVTOJHKJHDLCSXJWKEKIV6BA, seg_pos={0 0}, upload=509109]: Done uploading piece. err: <nil> Dec 29 20:15:02.167078173 (580h46m2.038161228s) [noise=true , address=45.91.116.9:4262 , piece_id=UV5XMT447DE6QTJYRYBD7466OPDQ3YQSIVQ4C2BSIO37OCZKGSHA, seg_pos={0 0}, upload=509109]: Done uploading piece. err: <nil> Dec 29 20:15:02.167162874 (580h46m2.038245929s) [seg_pos={0 0}, upload=509109]: Segment reached optimal threshold of 13 pieces. Dec 29 20:15:02.170200207 (580h46m2.041283262s) [noise=true , address=45.91.116.6:4975 , piece_id=YHNIUJWPFG4EC7FJYL43CCVFTKZ2GMKX5ANKEIHDMLAL2AGA7V3A, seg_pos={0 0}, upload=509109]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: read tcp 188.166.225.170:42964->45.91.116.6:4975: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 29 20:15:02.170215235 (580h46m2.04129829s) [noise=true , address=45.91.116.6:4975 , piece_id=YHNIUJWPFG4EC7FJYL43CCVFTKZ2GMKX5ANKEIHDMLAL2AGA7V3A, seg_pos={0 0}, upload=509109]: Long tail cancellation due to optimal threshold Dec 29 20:15:02.170672046 (580h46m2.041755101s) [noise=true , address=145.79.227.44:4194 , piece_id=NEQVB6ERVDONQCXWRFXXFPZ3EBSLRJJMWOZHSBCKZWFWZ5FJXTNA, seg_pos={0 0}, upload=509109]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh): protocol: expected piece hash; context canceled; EOF Dec 29 20:15:02.170683507 (580h46m2.041766553s) [noise=true , address=145.79.227.44:4194 , piece_id=NEQVB6ERVDONQCXWRFXXFPZ3EBSLRJJMWOZHSBCKZWFWZ5FJXTNA, seg_pos={0 0}, upload=509109]: Long tail cancellation due to optimal threshold Dec 29 20:15:02.170846298 (580h46m2.041929353s) [noise=true , address=145.79.227.28:4725 , piece_id=U442UUBRULRA3AY6H7EP5G2XJFY47YYDMBWRCMQBM2BLCKZJBYDA, seg_pos={0 0}, upload=509109]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12YKLTah3swX14kL82eNogJxd8D8Hjfq7SPjDDQAapDnXmDctP1): protocol: write tcp 188.166.225.170:56476->145.79.227.28:4725: use of closed network connection; write tcp 188.166.225.170:56476->145.79.227.28:4725: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56476->145.79.227.28:4725: use of closed network connection Dec 29 20:15:02.170859112 (580h46m2.041942157s) [noise=true , address=145.79.227.28:4725 , piece_id=U442UUBRULRA3AY6H7EP5G2XJFY47YYDMBWRCMQBM2BLCKZJBYDA, seg_pos={0 0}, upload=509109]: Long tail cancellation due to optimal threshold Dec 29 20:15:02.170884811 (580h46m2.041967856s) [seg_pos={0 0}, upload=509109]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 20:15:02.171053743 (580h46m2.042136788s) [upload=509109]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 20:15:02.171062129 (580h46m2.042145184s) [upload=509109]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 20:15:02.455142652 (580h46m2.326225707s) [upload=509109]: Done uploading segments... Dec 29 20:15:02.455171597 (580h46m2.326254642s) [upload=509109]: Upload finished. err: <nil> Duration: 28.737164223s core uplink us-select-1
checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing) antiblip: 1 out of 60 is failed (threshold=6) [...........................X................................] duration: 249.121907ms remote: "tcp:207.211.208.129:443" target: https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt trace: https://jaeger.dev.storj.io/trace/4679501fce62ca7 FAIL (2025-12-29 20:19:34 UTC) orbiter warning: Get "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt": context deadline exceeded Duration: 249.144108ms edge linksharing bgp eu1
checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing) antiblip: 1 out of 60 is failed (threshold=6) [............................X...............................] duration: 249.79598ms remote: "tcp:207.211.208.129:443" target: https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt trace: https://jaeger.dev.storj.io/trace/4679501fce62ca7 FAIL (2025-12-29 20:18:58 UTC) orbiter warning: Get "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt": context deadline exceeded Duration: 249.810707ms edge linksharing bgp eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 34.286135278s trace: https://jaeger.dev.storj.io/trace/24ac2b258db6f49 FAIL (2025-12-29 20:18:49 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 19.881883s) [noise=true , address=145.79.227.44:4213 , piece_id=47WPDBUPSA5QUYOBYSFQFQMIFZMJJTD3LBYL6J4WCM4G2LLYKRBQ, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.011913035 (580h4m19.88299608s) [noise=true , address=145.79.227.11:4748 , piece_id=3FFT5U7HDYRGKI67WTT3EWL6PSQY4MOFM2RVALUI2SXDQVFAUV7A, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.012333796 (580h4m19.883416831s) [noise=true , address=45.91.116.9:4283 , piece_id=W2IRSXBWJZBSADPBETO254LH5WH7RWHLP65FMYCW3QLTDZMWAJ2A, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.012776579 (580h4m19.883859614s) [noise=true , address=145.79.227.6:4283 , piece_id=UZISQ26DIRCHFWIKODJCEQYD2N3FEWFKC7CAL7LRB45KAGPHC5BQ, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.013135943 (580h4m19.884218978s) [noise=true , address=145.79.227.42:4150 , piece_id=TT4PDRJ6HRBNFYKWOBEHOPYSPMYVN2NZZKUBK3E5ACHRKJPK4ZMA, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.013460732 (580h4m19.884543767s) [noise=true , address=77.90.31.17:4786 , piece_id=EJTLQD7JFIXVQ4GZ6S35KOISBSCOVOCW44R7JDDN5HLWRXMZDQ6A, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.013853229 (580h4m19.884936264s) [noise=true , address=145.79.227.29:4959 , piece_id=IUPWCJOL4JXCAOUTP4GCQMK657U53VGQD4JXRNNHSYLK2AK56WNA, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.014144343 (580h4m19.885227388s) [noise=true , address=145.79.227.33:4893 , piece_id=QTATPQWRYUQ33FL2M3RAP762AM3AC4PT7BHBCIUT7GFNVDC5UDGA, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.014250846 (580h4m19.885333881s) [noise=true , address=5.231.73.4:4027 , piece_id=W56MVTNRZX4XFEMCJULZEX6GXQ4BNDNJ3R3EZ5LDHWVCGGMB2NTQ, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.014906585 (580h4m19.88598962s) [noise=true , address=145.79.227.30:4773 , piece_id=BAR4QMZ7OM74IIPLRGWAMEYZKIILXXQNRYJFBUHXV5YUWYD7X2YA, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.015511426 (580h4m19.886594471s) [noise=true , address=64.49.13.169:4711 , piece_id=MZYKDHA7E4KNOZ4C752DWBLJJPJQTBGCRX6QACQCCCIJ5VT4LU4Q, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.016659873 (580h4m19.887742918s) [noise=true , address=145.79.227.8:4335 , piece_id=Q7UEGDD4AGZREL2V2VRHEVDJFIDPKKUF6JCZ2FJGQYJAVW7KQAKQ, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.018312429 (580h4m19.889395464s) [noise=true , address=64.49.13.27:4966 , piece_id=GABX3GQG7TGY36WBSO3JWMBEIOK7R7MDZA4IHPPNBRBFXBTMJO3A, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.019851220 (580h4m19.890934255s) [noise=true , address=77.90.31.18:4836 , piece_id=ONCF7RXVICUL55HJ2AWB466YFRANSOIN4WKMH5VCURXP334CRG5A, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.021306711 (580h4m19.892389746s) [noise=true , address=145.79.227.34:4913 , piece_id=O22AXRPCITP6BCWH4ZANG2Z2F4PENTRZOCIA3YUMXYBVP7XXRWUA, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:21.467852993 (580h4m21.338936028s) [upload=508498]: Next returned nil segment Dec 29 19:33:21.467865737 (580h4m21.338948772s) [upload=508498]: Waiting for error group managing segments... Dec 29 19:33:21.864176322 (580h4m21.735259357s) [noise=true , address=145.79.227.11:4748 , piece_id=3FFT5U7HDYRGKI67WTT3EWL6PSQY4MOFM2RVALUI2SXDQVFAUV7A, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:21.890454300 (580h4m21.761537345s) [noise=true , address=145.79.227.8:4335 , piece_id=Q7UEGDD4AGZREL2V2VRHEVDJFIDPKKUF6JCZ2FJGQYJAVW7KQAKQ, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:21.895347667 (580h4m21.766430713s) [noise=true , address=145.79.227.42:4150 , piece_id=TT4PDRJ6HRBNFYKWOBEHOPYSPMYVN2NZZKUBK3E5ACHRKJPK4ZMA, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:21.912551226 (580h4m21.783634261s) [noise=true , address=145.79.227.29:4959 , piece_id=IUPWCJOL4JXCAOUTP4GCQMK657U53VGQD4JXRNNHSYLK2AK56WNA, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:21.918776128 (580h4m21.789859173s) [noise=true , address=145.79.227.6:4283 , piece_id=UZISQ26DIRCHFWIKODJCEQYD2N3FEWFKC7CAL7LRB45KAGPHC5BQ, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.049639643 (580h4m21.920722678s) [noise=true , address=5.231.73.4:4027 , piece_id=W56MVTNRZX4XFEMCJULZEX6GXQ4BNDNJ3R3EZ5LDHWVCGGMB2NTQ, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.062422904 (580h4m21.933505939s) [noise=true , address=77.90.31.17:4786 , piece_id=EJTLQD7JFIXVQ4GZ6S35KOISBSCOVOCW44R7JDDN5HLWRXMZDQ6A, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.087400355 (580h4m21.9584834s) [noise=true , address=77.90.31.18:4836 , piece_id=ONCF7RXVICUL55HJ2AWB466YFRANSOIN4WKMH5VCURXP334CRG5A, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.123477030 (580h4m21.994560075s) [noise=true , address=77.90.31.10:4542 , piece_id=77QBOT4PZXNTMK7Z4SNPYVSJDNKZAJYKZIE2XJO2KOKYNFS6XN6Q, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.147097236 (580h4m22.018180271s) [noise=true , address=77.90.31.10:4539 , piece_id=HFHV4GSLPWU2NXE24SCA7XHPSQMHEKCTT7HOPBXJYQDN5XYQBUTA, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.187114676 (580h4m22.058197721s) [noise=true , address=45.91.116.9:4283 , piece_id=W2IRSXBWJZBSADPBETO254LH5WH7RWHLP65FMYCW3QLTDZMWAJ2A, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.755107038 (580h4m22.626190073s) [noise=true , address=145.79.227.43:4190 , piece_id=5O4HKBMCZZ5FWQCKO2B3NJVBV2ZHAVZCYRYRPHCURU5ZQ2SPIE5A, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.779882074 (580h4m22.650965109s) [noise=true , address=145.79.227.30:4773 , piece_id=BAR4QMZ7OM74IIPLRGWAMEYZKIILXXQNRYJFBUHXV5YUWYD7X2YA, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.820831227 (580h4m22.691914262s) [noise=true , address=145.79.227.44:4213 , piece_id=47WPDBUPSA5QUYOBYSFQFQMIFZMJJTD3LBYL6J4WCM4G2LLYKRBQ, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.820854071 (580h4m22.691937106s) [seg_pos={0 0}, upload=508498]: Segment reached optimal threshold of 14 pieces. Dec 29 19:33:22.821045084 (580h4m22.692128119s) [noise=true , address=145.79.227.34:4913 , piece_id=O22AXRPCITP6BCWH4ZANG2Z2F4PENTRZOCIA3YUMXYBVP7XXRWUA, seg_pos={0 0}, upload=508498]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1228o9xxwySb7RCPH81yobfdRxNQoz82JYgkb9TEw8eeX7CATgC): protocol: expected piece hash; context canceled; EOF Dec 29 19:33:22.821051897 (580h4m22.692134932s) [noise=true , address=145.79.227.34:4913 , piece_id=O22AXRPCITP6BCWH4ZANG2Z2F4PENTRZOCIA3YUMXYBVP7XXRWUA, seg_pos={0 0}, upload=508498]: Long tail cancellation due to optimal threshold Dec 29 19:33:22.821168860 (580h4m22.692251905s) [noise=true , address=64.49.13.169:4711 , piece_id=MZYKDHA7E4KNOZ4C752DWBLJJPJQTBGCRX6QACQCCCIJ5VT4LU4Q, seg_pos={0 0}, upload=508498]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nxT1Ldr46HPrDRHQ2nrwEBZT5bEVoPAAfBz86ZYwUH13sqwHu): protocol: write tcp 188.166.225.170:52944->64.49.13.169:4711: use of closed network connection; write tcp 188.166.225.170:52944->64.49.13.169:4711: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52944->64.49.13.169:4711: use of closed network connection Dec 29 19:33:22.821176013 (580h4m22.692259049s) [noise=true , address=64.49.13.169:4711 , piece_id=MZYKDHA7E4KNOZ4C752DWBLJJPJQTBGCRX6QACQCCCIJ5VT4LU4Q, seg_pos={0 0}, upload=508498]: Long tail cancellation due to optimal threshold Dec 29 19:33:22.821193216 (580h4m22.692276251s) [noise=true , address=64.49.13.27:4966 , piece_id=GABX3GQG7TGY36WBSO3JWMBEIOK7R7MDZA4IHPPNBRBFXBTMJO3A, seg_pos={0 0}, upload=508498]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gUV64TP8cTQhBjswnhcWfQHwYEEWqr5ASuPjtEx3X6ZQNVhS5): protocol: write tcp 188.166.225.170:47300->64.49.13.27:4966: use of closed network connection; write tcp 188.166.225.170:47300->64.49.13.27:4966: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47300->64.49.13.27:4966: use of closed network connection Dec 29 19:33:22.821198947 (580h4m22.692281983s) [noise=true , address=64.49.13.27:4966 , piece_id=GABX3GQG7TGY36WBSO3JWMBEIOK7R7MDZA4IHPPNBRBFXBTMJO3A, seg_pos={0 0}, upload=508498]: Long tail cancellation due to optimal threshold Dec 29 19:33:22.821300189 (580h4m22.692383225s) [noise=true , address=145.79.227.33:4893 , piece_id=QTATPQWRYUQ33FL2M3RAP762AM3AC4PT7BHBCIUT7GFNVDC5UDGA, seg_pos={0 0}, upload=508498]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17VMFZ2T8jrSJVtchciJ9SnyrR6JstvcMoXrN5Q3zTD1aQU1oJ): protocol: expected piece hash; context canceled; EOF Dec 29 19:33:22.821306302 (580h4m22.692389337s) [noise=true , address=145.79.227.33:4893 , piece_id=QTATPQWRYUQ33FL2M3RAP762AM3AC4PT7BHBCIUT7GFNVDC5UDGA, seg_pos={0 0}, upload=508498]: Long tail cancellation due to optimal threshold Dec 29 19:33:22.821399579 (580h4m22.692482614s) [seg_pos={0 0}, upload=508498]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 19:33:22.821465755 (580h4m22.6925488s) [upload=508498]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 19:33:22.821470243 (580h4m22.692553278s) [upload=508498]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 19:33:23.123995567 (580h4m22.995078612s) [upload=508498]: Done uploading segments... Dec 29 19:33:23.124020665 (580h4m22.99510371s) [upload=508498]: Upload finished. err: <nil> Duration: 34.286164444s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing) antiblip: 1 out of 60 is failed (threshold=6) [.............................X..............................] duration: 251.382351ms remote: "tcp:207.211.208.129:443" target: https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt trace: https://jaeger.dev.storj.io/trace/4679501fce62ca7 FAIL (2025-12-29 20:18:37 UTC) orbiter warning: Get "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt": context deadline exceeded Duration: 251.405486ms edge linksharing bgp eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [XX.XX...X.X....X..X.X....X.......X.....................X....] duration: 6.419774623s trace: https://jaeger.dev.storj.io/trace/607cf963331dd581 FAIL (2025-12-29 20:18:34 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 20:14:59.343887631 (580h45m59.214970666s) [upload=509109]: Done starting upload Dec 29 20:14:59.343929450 (580h45m59.215012485s) [upload=509109]: Uploading segments... Dec 29 20:14:59.344178725 (580h45m59.21526176s) [upload=509109]: Got next segment. Inline: false Dec 29 20:14:59.344213341 (580h45m59.215296376s) [upload=509109]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 20:14:59.344218040 (580h45m59.215301075s) [upload=509109]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 20:14:59.620876250 (580h45m59.491959285s) [seg_pos={0 0}, upload=509109]: Begin upload segment... Dec 29 20:14:59.620932867 (580h45m59.492015912s) [seg_pos={0 0}, upload=509109]: Done begin upload segment. Dec 29 20:14:59.620975569 (580h45m59.492058604s) [noise=true , address=145.79.227.51:4411 , piece_id=LYCY2Z7Z3BFR5XSM6JXFCL5LZAU56PUBVGHLVURZUTQTEZF7CLFQ, seg_pos={0 0}, upload=509109]: Uploading piece... Dec 29 20:14:59.620983133 (580h45m59.492066178s) [noise=true , address=145.79.227.48:4303 , piece_id=UYEJZYH7QYLL6KGKEQBH6DEGEBMWZSM2OT7W4V6D4D3FQV4XUDZQ, seg_pos={0 0}, upload=509109]: Uploading piece... Dec 29 20:14:59.621128750 (580h45m59.492211796s) [noise=true , address=145.79.227.46:4602 , piece_id=ZJA75URLP6YW2ROXGLR4PONX2PXXWZNGGANAI37SREOPJ5S7OV3A, seg_pos={0 0}, upload=509109]: Uploading piece... Dec 29 20:14:59.621771063 (580h45m59.492854098s) [noise=true , address=145.79.227.15:4875 , piece_id=RFIPM3MTYFN5PLV4NVNF7AE5OVA54ZIOR2PUY7S4674CN466ECBA, seg_pos={0 0}, upload=509109]: Uploading piece... Dec 29 20:14:59.622036208 (580h45m59.493119263s) [noise=true , address=145.79.227.18:4006 , piece_id=QP3MYNJN7EGPAM7NB4E2ITEV6EPATVAIWGAPT3KNEUXEPPBNQVDA, seg_pos={0 0}, upload=509109]: Uploading piece... Dec 29 20:14:59.622303207 (580h45m59.493386252s) [noise=true , address=145.79.227.28:4725 , piece_id=U442UUBRULRA3AY6H7EP5G2XJFY47YYDMBWRCMQBM2BLCKZJBYDA, seg_pos={0 0}, upload=509109]: Uploading piece... Dec 29 20:14:59.622693610 (580h45m59.493776645s) [noise=true , address=145.79.227.44:4194 , piece_id=NEQVB6ERVDONQCXWRFXXFPZ3EBSLRJJMWOZHSBCKZWFWZ5FJXTNA, seg_pos={0 0}, upload=509109]: Uploading piece... Dec 29 20:14:59.623383705 (580h45m59.49446674s) [noise=true , address=77.90.31.13:4627 , piece_id=OVJXRA53CFJFZSRJ6H2MHZLO4RCLLC3TIGGYQN3ESNQFA3ESQD3A, seg_pos={0 0}, upload=509109]: Uploading piece... Dec 29 20:14:59.623912161 (580h45m59.494995206s) [noise=true , address=45.91.116.7:4010 , piece_id=BO772OMWJBZDCBAX3O6ECC766FEHZVTOJHKJHDLCSXJWKEKIV6BA, seg_pos={0 0}, upload=509109]: Uploading piece... Dec 29 20:14:59.624389991 (580h45m59.495473026s) [noise=true , address=145.79.227.2:4115 , piece_id=J6ZAJU7FYPAFEI47EJBUUUCASWZY2PZZWABSA7WRLAQPYQMFTENQ, seg_pos={0 0}, upload=509109]: Uploading piece... Dec 29 20:14:59.624497966 (580h45m59.495581001s) [noise=true , address=145.79.227.11:4731 , piece_id=46XM74D3UEXNZXHZYWFOYW6SHHWTQU2HA6Q2WYSJLH7OVC23JALQ, seg_pos={0 0}, upload=509109]: Uploading piece... Dec 29 20:14:59.624935829 (580h45m59.496018864s) [noise=true , address=145.79.227.19:4078 , piece_id=NZLVPFPYJXZHMK2XSIMIDBR7CN4OWVWT5QGNLTK3STC5KBY7CMGA, seg_pos={0 0}, upload=509109]: Uploading piece... Dec 29 20:14:59.624977129 (580h45m59.496060174s) [noise=true , address=45.91.116.9:4262 , piece_id=UV5XMT447DE6QTJYRYBD7466OPDQ3YQSIVQ4C2BSIO37OCZKGSHA, seg_pos={0 0}, upload=509109]: Uploading piece... Dec 29 20:14:59.625338717 (580h45m59.496421762s) [noise=true , address=45.91.116.6:4975 , piece_id=YHNIUJWPFG4EC7FJYL43CCVFTKZ2GMKX5ANKEIHDMLAL2AGA7V3A, seg_pos={0 0}, upload=509109]: Uploading piece... Dec 29 20:14:59.625486658 (580h45m59.496569704s) [noise=true , address=145.79.227.36:4518 , piece_id=3DYS7R2VTEUGCDWBWFP2OZHYTMQGLK7OS3OFSCUBIKWSWHJ44ELQ, seg_pos={0 0}, upload=509109]: Uploading piece... Dec 29 20:14:59.625852676 (580h45m59.496935721s) [noise=true , address=145.79.227.9:4366 , piece_id=ICYZM4QQZQJLS4T4KYM47NEMIKYD7PZWQ2U2NEFLAWLLPO6OFG3A, seg_pos={0 0}, upload=509109]: Uploading piece... Dec 29 20:15:01.223830613 (580h46m1.094913658s) [upload=509109]: Next returned nil segment Dec 29 20:15:01.223865600 (580h46m1.094948645s) [upload=509109]: Waiting for error group managing segments... Dec 29 20:15:01.599999537 (580h46m1.471082582s) [noise=true , address=145.79.227.11:4731 , piece_id=46XM74D3UEXNZXHZYWFOYW6SHHWTQU2HA6Q2WYSJLH7OVC23JALQ, seg_pos={0 0}, upload=509109]: Done uploading piece. err: <nil> Dec 29 20:15:01.606734191 (580h46m1.477817236s) [noise=true , address=145.79.227.9:4366 , piece_id=ICYZM4QQZQJLS4T4KYM47NEMIKYD7PZWQ2U2NEFLAWLLPO6OFG3A, seg_pos={0 0}, upload=509109]: Done uploading piece. err: <nil> Dec 29 20:15:01.609790219 (580h46m1.480873265s) [noise=true , address=145.79.227.48:4303 , piece_id=UYEJZYH7QYLL6KGKEQBH6DEGEBMWZSM2OT7W4V6D4D3FQV4XUDZQ, seg_pos={0 0}, upload=509109]: Done uploading piece. err: <nil> Dec 29 20:15:01.613856723 (580h46m1.484939768s) [noise=true , address=145.79.227.51:4411 , piece_id=LYCY2Z7Z3BFR5XSM6JXFCL5LZAU56PUBVGHLVURZUTQTEZF7CLFQ, seg_pos={0 0}, upload=509109]: Done uploading piece. err: <nil> Dec 29 20:15:01.615434938 (580h46m1.486517973s) [noise=true , address=145.79.227.46:4602 , piece_id=ZJA75URLP6YW2ROXGLR4PONX2PXXWZNGGANAI37SREOPJ5S7OV3A, seg_pos={0 0}, upload=509109]: Done uploading piece. err: <nil> Dec 29 20:15:01.617581145 (580h46m1.48866419s) [noise=true , address=145.79.227.19:4078 , piece_id=NZLVPFPYJXZHMK2XSIMIDBR7CN4OWVWT5QGNLTK3STC5KBY7CMGA, seg_pos={0 0}, upload=509109]: Done uploading piece. err: <nil> Dec 29 20:15:01.618039818 (580h46m1.489122863s) [noise=true , address=145.79.227.15:4875 , piece_id=RFIPM3MTYFN5PLV4NVNF7AE5OVA54ZIOR2PUY7S4674CN466ECBA, seg_pos={0 0}, upload=509109]: Done uploading piece. err: <nil> Dec 29 20:15:01.624408906 (580h46m1.495491951s) [noise=true , address=145.79.227.36:4518 , piece_id=3DYS7R2VTEUGCDWBWFP2OZHYTMQGLK7OS3OFSCUBIKWSWHJ44ELQ, seg_pos={0 0}, upload=509109]: Done uploading piece. err: <nil> Dec 29 20:15:01.629708417 (580h46m1.500791482s) [noise=true , address=145.79.227.2:4115 , piece_id=J6ZAJU7FYPAFEI47EJBUUUCASWZY2PZZWABSA7WRLAQPYQMFTENQ, seg_pos={0 0}, upload=509109]: Done uploading piece. err: <nil> Dec 29 20:15:01.654622637 (580h46m1.525705682s) [noise=true , address=145.79.227.18:4006 , piece_id=QP3MYNJN7EGPAM7NB4E2ITEV6EPATVAIWGAPT3KNEUXEPPBNQVDA, seg_pos={0 0}, upload=509109]: Done uploading piece. err: <nil> Dec 29 20:15:01.799321647 (580h46m1.670404682s) [noise=true , address=77.90.31.13:4627 , piece_id=OVJXRA53CFJFZSRJ6H2MHZLO4RCLLC3TIGGYQN3ESNQFA3ESQD3A, seg_pos={0 0}, upload=509109]: Done uploading piece. err: <nil> Dec 29 20:15:01.902753888 (580h46m1.773836943s) [noise=true , address=45.91.116.7:4010 , piece_id=BO772OMWJBZDCBAX3O6ECC766FEHZVTOJHKJHDLCSXJWKEKIV6BA, seg_pos={0 0}, upload=509109]: Done uploading piece. err: <nil> Dec 29 20:15:02.167078173 (580h46m2.038161228s) [noise=true , address=45.91.116.9:4262 , piece_id=UV5XMT447DE6QTJYRYBD7466OPDQ3YQSIVQ4C2BSIO37OCZKGSHA, seg_pos={0 0}, upload=509109]: Done uploading piece. err: <nil> Dec 29 20:15:02.167162874 (580h46m2.038245929s) [seg_pos={0 0}, upload=509109]: Segment reached optimal threshold of 13 pieces. Dec 29 20:15:02.170200207 (580h46m2.041283262s) [noise=true , address=45.91.116.6:4975 , piece_id=YHNIUJWPFG4EC7FJYL43CCVFTKZ2GMKX5ANKEIHDMLAL2AGA7V3A, seg_pos={0 0}, upload=509109]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: read tcp 188.166.225.170:42964->45.91.116.6:4975: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 29 20:15:02.170215235 (580h46m2.04129829s) [noise=true , address=45.91.116.6:4975 , piece_id=YHNIUJWPFG4EC7FJYL43CCVFTKZ2GMKX5ANKEIHDMLAL2AGA7V3A, seg_pos={0 0}, upload=509109]: Long tail cancellation due to optimal threshold Dec 29 20:15:02.170672046 (580h46m2.041755101s) [noise=true , address=145.79.227.44:4194 , piece_id=NEQVB6ERVDONQCXWRFXXFPZ3EBSLRJJMWOZHSBCKZWFWZ5FJXTNA, seg_pos={0 0}, upload=509109]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh): protocol: expected piece hash; context canceled; EOF Dec 29 20:15:02.170683507 (580h46m2.041766553s) [noise=true , address=145.79.227.44:4194 , piece_id=NEQVB6ERVDONQCXWRFXXFPZ3EBSLRJJMWOZHSBCKZWFWZ5FJXTNA, seg_pos={0 0}, upload=509109]: Long tail cancellation due to optimal threshold Dec 29 20:15:02.170846298 (580h46m2.041929353s) [noise=true , address=145.79.227.28:4725 , piece_id=U442UUBRULRA3AY6H7EP5G2XJFY47YYDMBWRCMQBM2BLCKZJBYDA, seg_pos={0 0}, upload=509109]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12YKLTah3swX14kL82eNogJxd8D8Hjfq7SPjDDQAapDnXmDctP1): protocol: write tcp 188.166.225.170:56476->145.79.227.28:4725: use of closed network connection; write tcp 188.166.225.170:56476->145.79.227.28:4725: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56476->145.79.227.28:4725: use of closed network connection Dec 29 20:15:02.170859112 (580h46m2.041942157s) [noise=true , address=145.79.227.28:4725 , piece_id=U442UUBRULRA3AY6H7EP5G2XJFY47YYDMBWRCMQBM2BLCKZJBYDA, seg_pos={0 0}, upload=509109]: Long tail cancellation due to optimal threshold Dec 29 20:15:02.170884811 (580h46m2.041967856s) [seg_pos={0 0}, upload=509109]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 20:15:02.171053743 (580h46m2.042136788s) [upload=509109]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 20:15:02.171062129 (580h46m2.042145184s) [upload=509109]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 20:15:02.455142652 (580h46m2.326225707s) [upload=509109]: Done uploading segments... Dec 29 20:15:02.455171597 (580h46m2.326254642s) [upload=509109]: Upload finished. err: <nil> Duration: 6.419816903s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 18.460015153s trace: https://jaeger.dev.storj.io/trace/24ac2b258db6f49 FAIL (2025-12-29 20:18:03 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 19.881883s) [noise=true , address=145.79.227.44:4213 , piece_id=47WPDBUPSA5QUYOBYSFQFQMIFZMJJTD3LBYL6J4WCM4G2LLYKRBQ, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.011913035 (580h4m19.88299608s) [noise=true , address=145.79.227.11:4748 , piece_id=3FFT5U7HDYRGKI67WTT3EWL6PSQY4MOFM2RVALUI2SXDQVFAUV7A, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.012333796 (580h4m19.883416831s) [noise=true , address=45.91.116.9:4283 , piece_id=W2IRSXBWJZBSADPBETO254LH5WH7RWHLP65FMYCW3QLTDZMWAJ2A, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.012776579 (580h4m19.883859614s) [noise=true , address=145.79.227.6:4283 , piece_id=UZISQ26DIRCHFWIKODJCEQYD2N3FEWFKC7CAL7LRB45KAGPHC5BQ, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.013135943 (580h4m19.884218978s) [noise=true , address=145.79.227.42:4150 , piece_id=TT4PDRJ6HRBNFYKWOBEHOPYSPMYVN2NZZKUBK3E5ACHRKJPK4ZMA, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.013460732 (580h4m19.884543767s) [noise=true , address=77.90.31.17:4786 , piece_id=EJTLQD7JFIXVQ4GZ6S35KOISBSCOVOCW44R7JDDN5HLWRXMZDQ6A, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.013853229 (580h4m19.884936264s) [noise=true , address=145.79.227.29:4959 , piece_id=IUPWCJOL4JXCAOUTP4GCQMK657U53VGQD4JXRNNHSYLK2AK56WNA, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.014144343 (580h4m19.885227388s) [noise=true , address=145.79.227.33:4893 , piece_id=QTATPQWRYUQ33FL2M3RAP762AM3AC4PT7BHBCIUT7GFNVDC5UDGA, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.014250846 (580h4m19.885333881s) [noise=true , address=5.231.73.4:4027 , piece_id=W56MVTNRZX4XFEMCJULZEX6GXQ4BNDNJ3R3EZ5LDHWVCGGMB2NTQ, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.014906585 (580h4m19.88598962s) [noise=true , address=145.79.227.30:4773 , piece_id=BAR4QMZ7OM74IIPLRGWAMEYZKIILXXQNRYJFBUHXV5YUWYD7X2YA, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.015511426 (580h4m19.886594471s) [noise=true , address=64.49.13.169:4711 , piece_id=MZYKDHA7E4KNOZ4C752DWBLJJPJQTBGCRX6QACQCCCIJ5VT4LU4Q, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.016659873 (580h4m19.887742918s) [noise=true , address=145.79.227.8:4335 , piece_id=Q7UEGDD4AGZREL2V2VRHEVDJFIDPKKUF6JCZ2FJGQYJAVW7KQAKQ, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.018312429 (580h4m19.889395464s) [noise=true , address=64.49.13.27:4966 , piece_id=GABX3GQG7TGY36WBSO3JWMBEIOK7R7MDZA4IHPPNBRBFXBTMJO3A, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.019851220 (580h4m19.890934255s) [noise=true , address=77.90.31.18:4836 , piece_id=ONCF7RXVICUL55HJ2AWB466YFRANSOIN4WKMH5VCURXP334CRG5A, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.021306711 (580h4m19.892389746s) [noise=true , address=145.79.227.34:4913 , piece_id=O22AXRPCITP6BCWH4ZANG2Z2F4PENTRZOCIA3YUMXYBVP7XXRWUA, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:21.467852993 (580h4m21.338936028s) [upload=508498]: Next returned nil segment Dec 29 19:33:21.467865737 (580h4m21.338948772s) [upload=508498]: Waiting for error group managing segments... Dec 29 19:33:21.864176322 (580h4m21.735259357s) [noise=true , address=145.79.227.11:4748 , piece_id=3FFT5U7HDYRGKI67WTT3EWL6PSQY4MOFM2RVALUI2SXDQVFAUV7A, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:21.890454300 (580h4m21.761537345s) [noise=true , address=145.79.227.8:4335 , piece_id=Q7UEGDD4AGZREL2V2VRHEVDJFIDPKKUF6JCZ2FJGQYJAVW7KQAKQ, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:21.895347667 (580h4m21.766430713s) [noise=true , address=145.79.227.42:4150 , piece_id=TT4PDRJ6HRBNFYKWOBEHOPYSPMYVN2NZZKUBK3E5ACHRKJPK4ZMA, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:21.912551226 (580h4m21.783634261s) [noise=true , address=145.79.227.29:4959 , piece_id=IUPWCJOL4JXCAOUTP4GCQMK657U53VGQD4JXRNNHSYLK2AK56WNA, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:21.918776128 (580h4m21.789859173s) [noise=true , address=145.79.227.6:4283 , piece_id=UZISQ26DIRCHFWIKODJCEQYD2N3FEWFKC7CAL7LRB45KAGPHC5BQ, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.049639643 (580h4m21.920722678s) [noise=true , address=5.231.73.4:4027 , piece_id=W56MVTNRZX4XFEMCJULZEX6GXQ4BNDNJ3R3EZ5LDHWVCGGMB2NTQ, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.062422904 (580h4m21.933505939s) [noise=true , address=77.90.31.17:4786 , piece_id=EJTLQD7JFIXVQ4GZ6S35KOISBSCOVOCW44R7JDDN5HLWRXMZDQ6A, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.087400355 (580h4m21.9584834s) [noise=true , address=77.90.31.18:4836 , piece_id=ONCF7RXVICUL55HJ2AWB466YFRANSOIN4WKMH5VCURXP334CRG5A, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.123477030 (580h4m21.994560075s) [noise=true , address=77.90.31.10:4542 , piece_id=77QBOT4PZXNTMK7Z4SNPYVSJDNKZAJYKZIE2XJO2KOKYNFS6XN6Q, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.147097236 (580h4m22.018180271s) [noise=true , address=77.90.31.10:4539 , piece_id=HFHV4GSLPWU2NXE24SCA7XHPSQMHEKCTT7HOPBXJYQDN5XYQBUTA, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.187114676 (580h4m22.058197721s) [noise=true , address=45.91.116.9:4283 , piece_id=W2IRSXBWJZBSADPBETO254LH5WH7RWHLP65FMYCW3QLTDZMWAJ2A, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.755107038 (580h4m22.626190073s) [noise=true , address=145.79.227.43:4190 , piece_id=5O4HKBMCZZ5FWQCKO2B3NJVBV2ZHAVZCYRYRPHCURU5ZQ2SPIE5A, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.779882074 (580h4m22.650965109s) [noise=true , address=145.79.227.30:4773 , piece_id=BAR4QMZ7OM74IIPLRGWAMEYZKIILXXQNRYJFBUHXV5YUWYD7X2YA, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.820831227 (580h4m22.691914262s) [noise=true , address=145.79.227.44:4213 , piece_id=47WPDBUPSA5QUYOBYSFQFQMIFZMJJTD3LBYL6J4WCM4G2LLYKRBQ, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.820854071 (580h4m22.691937106s) [seg_pos={0 0}, upload=508498]: Segment reached optimal threshold of 14 pieces. Dec 29 19:33:22.821045084 (580h4m22.692128119s) [noise=true , address=145.79.227.34:4913 , piece_id=O22AXRPCITP6BCWH4ZANG2Z2F4PENTRZOCIA3YUMXYBVP7XXRWUA, seg_pos={0 0}, upload=508498]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1228o9xxwySb7RCPH81yobfdRxNQoz82JYgkb9TEw8eeX7CATgC): protocol: expected piece hash; context canceled; EOF Dec 29 19:33:22.821051897 (580h4m22.692134932s) [noise=true , address=145.79.227.34:4913 , piece_id=O22AXRPCITP6BCWH4ZANG2Z2F4PENTRZOCIA3YUMXYBVP7XXRWUA, seg_pos={0 0}, upload=508498]: Long tail cancellation due to optimal threshold Dec 29 19:33:22.821168860 (580h4m22.692251905s) [noise=true , address=64.49.13.169:4711 , piece_id=MZYKDHA7E4KNOZ4C752DWBLJJPJQTBGCRX6QACQCCCIJ5VT4LU4Q, seg_pos={0 0}, upload=508498]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nxT1Ldr46HPrDRHQ2nrwEBZT5bEVoPAAfBz86ZYwUH13sqwHu): protocol: write tcp 188.166.225.170:52944->64.49.13.169:4711: use of closed network connection; write tcp 188.166.225.170:52944->64.49.13.169:4711: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52944->64.49.13.169:4711: use of closed network connection Dec 29 19:33:22.821176013 (580h4m22.692259049s) [noise=true , address=64.49.13.169:4711 , piece_id=MZYKDHA7E4KNOZ4C752DWBLJJPJQTBGCRX6QACQCCCIJ5VT4LU4Q, seg_pos={0 0}, upload=508498]: Long tail cancellation due to optimal threshold Dec 29 19:33:22.821193216 (580h4m22.692276251s) [noise=true , address=64.49.13.27:4966 , piece_id=GABX3GQG7TGY36WBSO3JWMBEIOK7R7MDZA4IHPPNBRBFXBTMJO3A, seg_pos={0 0}, upload=508498]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gUV64TP8cTQhBjswnhcWfQHwYEEWqr5ASuPjtEx3X6ZQNVhS5): protocol: write tcp 188.166.225.170:47300->64.49.13.27:4966: use of closed network connection; write tcp 188.166.225.170:47300->64.49.13.27:4966: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47300->64.49.13.27:4966: use of closed network connection Dec 29 19:33:22.821198947 (580h4m22.692281983s) [noise=true , address=64.49.13.27:4966 , piece_id=GABX3GQG7TGY36WBSO3JWMBEIOK7R7MDZA4IHPPNBRBFXBTMJO3A, seg_pos={0 0}, upload=508498]: Long tail cancellation due to optimal threshold Dec 29 19:33:22.821300189 (580h4m22.692383225s) [noise=true , address=145.79.227.33:4893 , piece_id=QTATPQWRYUQ33FL2M3RAP762AM3AC4PT7BHBCIUT7GFNVDC5UDGA, seg_pos={0 0}, upload=508498]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17VMFZ2T8jrSJVtchciJ9SnyrR6JstvcMoXrN5Q3zTD1aQU1oJ): protocol: expected piece hash; context canceled; EOF Dec 29 19:33:22.821306302 (580h4m22.692389337s) [noise=true , address=145.79.227.33:4893 , piece_id=QTATPQWRYUQ33FL2M3RAP762AM3AC4PT7BHBCIUT7GFNVDC5UDGA, seg_pos={0 0}, upload=508498]: Long tail cancellation due to optimal threshold Dec 29 19:33:22.821399579 (580h4m22.692482614s) [seg_pos={0 0}, upload=508498]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 19:33:22.821465755 (580h4m22.6925488s) [upload=508498]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 19:33:22.821470243 (580h4m22.692553278s) [upload=508498]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 19:33:23.123995567 (580h4m22.995078612s) [upload=508498]: Done uploading segments... Dec 29 19:33:23.124020665 (580h4m22.99510371s) [upload=508498]: Upload finished. err: <nil> Duration: 18.460043527s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [XXX.XX...X.X....X..X.X....X.......X.....................X...] duration: 7.681459059s trace: https://jaeger.dev.storj.io/trace/607cf963331dd581 FAIL (2025-12-29 20:18:02 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 20:14:59.343887631 (580h45m59.214970666s) [upload=509109]: Done starting upload Dec 29 20:14:59.343929450 (580h45m59.215012485s) [upload=509109]: Uploading segments... Dec 29 20:14:59.344178725 (580h45m59.21526176s) [upload=509109]: Got next segment. Inline: false Dec 29 20:14:59.344213341 (580h45m59.215296376s) [upload=509109]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 20:14:59.344218040 (580h45m59.215301075s) [upload=509109]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 20:14:59.620876250 (580h45m59.491959285s) [seg_pos={0 0}, upload=509109]: Begin upload segment... Dec 29 20:14:59.620932867 (580h45m59.492015912s) [seg_pos={0 0}, upload=509109]: Done begin upload segment. Dec 29 20:14:59.620975569 (580h45m59.492058604s) [noise=true , address=145.79.227.51:4411 , piece_id=LYCY2Z7Z3BFR5XSM6JXFCL5LZAU56PUBVGHLVURZUTQTEZF7CLFQ, seg_pos={0 0}, upload=509109]: Uploading piece... Dec 29 20:14:59.620983133 (580h45m59.492066178s) [noise=true , address=145.79.227.48:4303 , piece_id=UYEJZYH7QYLL6KGKEQBH6DEGEBMWZSM2OT7W4V6D4D3FQV4XUDZQ, seg_pos={0 0}, upload=509109]: Uploading piece... Dec 29 20:14:59.621128750 (580h45m59.492211796s) [noise=true , address=145.79.227.46:4602 , piece_id=ZJA75URLP6YW2ROXGLR4PONX2PXXWZNGGANAI37SREOPJ5S7OV3A, seg_pos={0 0}, upload=509109]: Uploading piece... Dec 29 20:14:59.621771063 (580h45m59.492854098s) [noise=true , address=145.79.227.15:4875 , piece_id=RFIPM3MTYFN5PLV4NVNF7AE5OVA54ZIOR2PUY7S4674CN466ECBA, seg_pos={0 0}, upload=509109]: Uploading piece... Dec 29 20:14:59.622036208 (580h45m59.493119263s) [noise=true , address=145.79.227.18:4006 , piece_id=QP3MYNJN7EGPAM7NB4E2ITEV6EPATVAIWGAPT3KNEUXEPPBNQVDA, seg_pos={0 0}, upload=509109]: Uploading piece... Dec 29 20:14:59.622303207 (580h45m59.493386252s) [noise=true , address=145.79.227.28:4725 , piece_id=U442UUBRULRA3AY6H7EP5G2XJFY47YYDMBWRCMQBM2BLCKZJBYDA, seg_pos={0 0}, upload=509109]: Uploading piece... Dec 29 20:14:59.622693610 (580h45m59.493776645s) [noise=true , address=145.79.227.44:4194 , piece_id=NEQVB6ERVDONQCXWRFXXFPZ3EBSLRJJMWOZHSBCKZWFWZ5FJXTNA, seg_pos={0 0}, upload=509109]: Uploading piece... Dec 29 20:14:59.623383705 (580h45m59.49446674s) [noise=true , address=77.90.31.13:4627 , piece_id=OVJXRA53CFJFZSRJ6H2MHZLO4RCLLC3TIGGYQN3ESNQFA3ESQD3A, seg_pos={0 0}, upload=509109]: Uploading piece... Dec 29 20:14:59.623912161 (580h45m59.494995206s) [noise=true , address=45.91.116.7:4010 , piece_id=BO772OMWJBZDCBAX3O6ECC766FEHZVTOJHKJHDLCSXJWKEKIV6BA, seg_pos={0 0}, upload=509109]: Uploading piece... Dec 29 20:14:59.624389991 (580h45m59.495473026s) [noise=true , address=145.79.227.2:4115 , piece_id=J6ZAJU7FYPAFEI47EJBUUUCASWZY2PZZWABSA7WRLAQPYQMFTENQ, seg_pos={0 0}, upload=509109]: Uploading piece... Dec 29 20:14:59.624497966 (580h45m59.495581001s) [noise=true , address=145.79.227.11:4731 , piece_id=46XM74D3UEXNZXHZYWFOYW6SHHWTQU2HA6Q2WYSJLH7OVC23JALQ, seg_pos={0 0}, upload=509109]: Uploading piece... Dec 29 20:14:59.624935829 (580h45m59.496018864s) [noise=true , address=145.79.227.19:4078 , piece_id=NZLVPFPYJXZHMK2XSIMIDBR7CN4OWVWT5QGNLTK3STC5KBY7CMGA, seg_pos={0 0}, upload=509109]: Uploading piece... Dec 29 20:14:59.624977129 (580h45m59.496060174s) [noise=true , address=45.91.116.9:4262 , piece_id=UV5XMT447DE6QTJYRYBD7466OPDQ3YQSIVQ4C2BSIO37OCZKGSHA, seg_pos={0 0}, upload=509109]: Uploading piece... Dec 29 20:14:59.625338717 (580h45m59.496421762s) [noise=true , address=45.91.116.6:4975 , piece_id=YHNIUJWPFG4EC7FJYL43CCVFTKZ2GMKX5ANKEIHDMLAL2AGA7V3A, seg_pos={0 0}, upload=509109]: Uploading piece... Dec 29 20:14:59.625486658 (580h45m59.496569704s) [noise=true , address=145.79.227.36:4518 , piece_id=3DYS7R2VTEUGCDWBWFP2OZHYTMQGLK7OS3OFSCUBIKWSWHJ44ELQ, seg_pos={0 0}, upload=509109]: Uploading piece... Dec 29 20:14:59.625852676 (580h45m59.496935721s) [noise=true , address=145.79.227.9:4366 , piece_id=ICYZM4QQZQJLS4T4KYM47NEMIKYD7PZWQ2U2NEFLAWLLPO6OFG3A, seg_pos={0 0}, upload=509109]: Uploading piece... Dec 29 20:15:01.223830613 (580h46m1.094913658s) [upload=509109]: Next returned nil segment Dec 29 20:15:01.223865600 (580h46m1.094948645s) [upload=509109]: Waiting for error group managing segments... Dec 29 20:15:01.599999537 (580h46m1.471082582s) [noise=true , address=145.79.227.11:4731 , piece_id=46XM74D3UEXNZXHZYWFOYW6SHHWTQU2HA6Q2WYSJLH7OVC23JALQ, seg_pos={0 0}, upload=509109]: Done uploading piece. err: <nil> Dec 29 20:15:01.606734191 (580h46m1.477817236s) [noise=true , address=145.79.227.9:4366 , piece_id=ICYZM4QQZQJLS4T4KYM47NEMIKYD7PZWQ2U2NEFLAWLLPO6OFG3A, seg_pos={0 0}, upload=509109]: Done uploading piece. err: <nil> Dec 29 20:15:01.609790219 (580h46m1.480873265s) [noise=true , address=145.79.227.48:4303 , piece_id=UYEJZYH7QYLL6KGKEQBH6DEGEBMWZSM2OT7W4V6D4D3FQV4XUDZQ, seg_pos={0 0}, upload=509109]: Done uploading piece. err: <nil> Dec 29 20:15:01.613856723 (580h46m1.484939768s) [noise=true , address=145.79.227.51:4411 , piece_id=LYCY2Z7Z3BFR5XSM6JXFCL5LZAU56PUBVGHLVURZUTQTEZF7CLFQ, seg_pos={0 0}, upload=509109]: Done uploading piece. err: <nil> Dec 29 20:15:01.615434938 (580h46m1.486517973s) [noise=true , address=145.79.227.46:4602 , piece_id=ZJA75URLP6YW2ROXGLR4PONX2PXXWZNGGANAI37SREOPJ5S7OV3A, seg_pos={0 0}, upload=509109]: Done uploading piece. err: <nil> Dec 29 20:15:01.617581145 (580h46m1.48866419s) [noise=true , address=145.79.227.19:4078 , piece_id=NZLVPFPYJXZHMK2XSIMIDBR7CN4OWVWT5QGNLTK3STC5KBY7CMGA, seg_pos={0 0}, upload=509109]: Done uploading piece. err: <nil> Dec 29 20:15:01.618039818 (580h46m1.489122863s) [noise=true , address=145.79.227.15:4875 , piece_id=RFIPM3MTYFN5PLV4NVNF7AE5OVA54ZIOR2PUY7S4674CN466ECBA, seg_pos={0 0}, upload=509109]: Done uploading piece. err: <nil> Dec 29 20:15:01.624408906 (580h46m1.495491951s) [noise=true , address=145.79.227.36:4518 , piece_id=3DYS7R2VTEUGCDWBWFP2OZHYTMQGLK7OS3OFSCUBIKWSWHJ44ELQ, seg_pos={0 0}, upload=509109]: Done uploading piece. err: <nil> Dec 29 20:15:01.629708417 (580h46m1.500791482s) [noise=true , address=145.79.227.2:4115 , piece_id=J6ZAJU7FYPAFEI47EJBUUUCASWZY2PZZWABSA7WRLAQPYQMFTENQ, seg_pos={0 0}, upload=509109]: Done uploading piece. err: <nil> Dec 29 20:15:01.654622637 (580h46m1.525705682s) [noise=true , address=145.79.227.18:4006 , piece_id=QP3MYNJN7EGPAM7NB4E2ITEV6EPATVAIWGAPT3KNEUXEPPBNQVDA, seg_pos={0 0}, upload=509109]: Done uploading piece. err: <nil> Dec 29 20:15:01.799321647 (580h46m1.670404682s) [noise=true , address=77.90.31.13:4627 , piece_id=OVJXRA53CFJFZSRJ6H2MHZLO4RCLLC3TIGGYQN3ESNQFA3ESQD3A, seg_pos={0 0}, upload=509109]: Done uploading piece. err: <nil> Dec 29 20:15:01.902753888 (580h46m1.773836943s) [noise=true , address=45.91.116.7:4010 , piece_id=BO772OMWJBZDCBAX3O6ECC766FEHZVTOJHKJHDLCSXJWKEKIV6BA, seg_pos={0 0}, upload=509109]: Done uploading piece. err: <nil> Dec 29 20:15:02.167078173 (580h46m2.038161228s) [noise=true , address=45.91.116.9:4262 , piece_id=UV5XMT447DE6QTJYRYBD7466OPDQ3YQSIVQ4C2BSIO37OCZKGSHA, seg_pos={0 0}, upload=509109]: Done uploading piece. err: <nil> Dec 29 20:15:02.167162874 (580h46m2.038245929s) [seg_pos={0 0}, upload=509109]: Segment reached optimal threshold of 13 pieces. Dec 29 20:15:02.170200207 (580h46m2.041283262s) [noise=true , address=45.91.116.6:4975 , piece_id=YHNIUJWPFG4EC7FJYL43CCVFTKZ2GMKX5ANKEIHDMLAL2AGA7V3A, seg_pos={0 0}, upload=509109]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: read tcp 188.166.225.170:42964->45.91.116.6:4975: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 29 20:15:02.170215235 (580h46m2.04129829s) [noise=true , address=45.91.116.6:4975 , piece_id=YHNIUJWPFG4EC7FJYL43CCVFTKZ2GMKX5ANKEIHDMLAL2AGA7V3A, seg_pos={0 0}, upload=509109]: Long tail cancellation due to optimal threshold Dec 29 20:15:02.170672046 (580h46m2.041755101s) [noise=true , address=145.79.227.44:4194 , piece_id=NEQVB6ERVDONQCXWRFXXFPZ3EBSLRJJMWOZHSBCKZWFWZ5FJXTNA, seg_pos={0 0}, upload=509109]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh): protocol: expected piece hash; context canceled; EOF Dec 29 20:15:02.170683507 (580h46m2.041766553s) [noise=true , address=145.79.227.44:4194 , piece_id=NEQVB6ERVDONQCXWRFXXFPZ3EBSLRJJMWOZHSBCKZWFWZ5FJXTNA, seg_pos={0 0}, upload=509109]: Long tail cancellation due to optimal threshold Dec 29 20:15:02.170846298 (580h46m2.041929353s) [noise=true , address=145.79.227.28:4725 , piece_id=U442UUBRULRA3AY6H7EP5G2XJFY47YYDMBWRCMQBM2BLCKZJBYDA, seg_pos={0 0}, upload=509109]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12YKLTah3swX14kL82eNogJxd8D8Hjfq7SPjDDQAapDnXmDctP1): protocol: write tcp 188.166.225.170:56476->145.79.227.28:4725: use of closed network connection; write tcp 188.166.225.170:56476->145.79.227.28:4725: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56476->145.79.227.28:4725: use of closed network connection Dec 29 20:15:02.170859112 (580h46m2.041942157s) [noise=true , address=145.79.227.28:4725 , piece_id=U442UUBRULRA3AY6H7EP5G2XJFY47YYDMBWRCMQBM2BLCKZJBYDA, seg_pos={0 0}, upload=509109]: Long tail cancellation due to optimal threshold Dec 29 20:15:02.170884811 (580h46m2.041967856s) [seg_pos={0 0}, upload=509109]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 20:15:02.171053743 (580h46m2.042136788s) [upload=509109]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 20:15:02.171062129 (580h46m2.042145184s) [upload=509109]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 20:15:02.455142652 (580h46m2.326225707s) [upload=509109]: Done uploading segments... Dec 29 20:15:02.455171597 (580h46m2.326254642s) [upload=509109]: Upload finished. err: <nil> Duration: 7.681493283s core uplink us-select-1
checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing) antiblip: 1 out of 60 is failed (threshold=6) [..............................X.............................] duration: 240.707614ms remote: "tcp:207.211.208.129:443" target: https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt trace: https://jaeger.dev.storj.io/trace/4679501fce62ca7 FAIL (2025-12-29 20:17:58 UTC) orbiter warning: Get "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt": context deadline exceeded Duration: 240.727252ms edge linksharing bgp eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 16.884058718s trace: https://jaeger.dev.storj.io/trace/24ac2b258db6f49 FAIL (2025-12-29 20:17:15 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 19.881883s) [noise=true , address=145.79.227.44:4213 , piece_id=47WPDBUPSA5QUYOBYSFQFQMIFZMJJTD3LBYL6J4WCM4G2LLYKRBQ, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.011913035 (580h4m19.88299608s) [noise=true , address=145.79.227.11:4748 , piece_id=3FFT5U7HDYRGKI67WTT3EWL6PSQY4MOFM2RVALUI2SXDQVFAUV7A, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.012333796 (580h4m19.883416831s) [noise=true , address=45.91.116.9:4283 , piece_id=W2IRSXBWJZBSADPBETO254LH5WH7RWHLP65FMYCW3QLTDZMWAJ2A, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.012776579 (580h4m19.883859614s) [noise=true , address=145.79.227.6:4283 , piece_id=UZISQ26DIRCHFWIKODJCEQYD2N3FEWFKC7CAL7LRB45KAGPHC5BQ, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.013135943 (580h4m19.884218978s) [noise=true , address=145.79.227.42:4150 , piece_id=TT4PDRJ6HRBNFYKWOBEHOPYSPMYVN2NZZKUBK3E5ACHRKJPK4ZMA, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.013460732 (580h4m19.884543767s) [noise=true , address=77.90.31.17:4786 , piece_id=EJTLQD7JFIXVQ4GZ6S35KOISBSCOVOCW44R7JDDN5HLWRXMZDQ6A, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.013853229 (580h4m19.884936264s) [noise=true , address=145.79.227.29:4959 , piece_id=IUPWCJOL4JXCAOUTP4GCQMK657U53VGQD4JXRNNHSYLK2AK56WNA, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.014144343 (580h4m19.885227388s) [noise=true , address=145.79.227.33:4893 , piece_id=QTATPQWRYUQ33FL2M3RAP762AM3AC4PT7BHBCIUT7GFNVDC5UDGA, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.014250846 (580h4m19.885333881s) [noise=true , address=5.231.73.4:4027 , piece_id=W56MVTNRZX4XFEMCJULZEX6GXQ4BNDNJ3R3EZ5LDHWVCGGMB2NTQ, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.014906585 (580h4m19.88598962s) [noise=true , address=145.79.227.30:4773 , piece_id=BAR4QMZ7OM74IIPLRGWAMEYZKIILXXQNRYJFBUHXV5YUWYD7X2YA, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.015511426 (580h4m19.886594471s) [noise=true , address=64.49.13.169:4711 , piece_id=MZYKDHA7E4KNOZ4C752DWBLJJPJQTBGCRX6QACQCCCIJ5VT4LU4Q, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.016659873 (580h4m19.887742918s) [noise=true , address=145.79.227.8:4335 , piece_id=Q7UEGDD4AGZREL2V2VRHEVDJFIDPKKUF6JCZ2FJGQYJAVW7KQAKQ, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.018312429 (580h4m19.889395464s) [noise=true , address=64.49.13.27:4966 , piece_id=GABX3GQG7TGY36WBSO3JWMBEIOK7R7MDZA4IHPPNBRBFXBTMJO3A, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.019851220 (580h4m19.890934255s) [noise=true , address=77.90.31.18:4836 , piece_id=ONCF7RXVICUL55HJ2AWB466YFRANSOIN4WKMH5VCURXP334CRG5A, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.021306711 (580h4m19.892389746s) [noise=true , address=145.79.227.34:4913 , piece_id=O22AXRPCITP6BCWH4ZANG2Z2F4PENTRZOCIA3YUMXYBVP7XXRWUA, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:21.467852993 (580h4m21.338936028s) [upload=508498]: Next returned nil segment Dec 29 19:33:21.467865737 (580h4m21.338948772s) [upload=508498]: Waiting for error group managing segments... Dec 29 19:33:21.864176322 (580h4m21.735259357s) [noise=true , address=145.79.227.11:4748 , piece_id=3FFT5U7HDYRGKI67WTT3EWL6PSQY4MOFM2RVALUI2SXDQVFAUV7A, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:21.890454300 (580h4m21.761537345s) [noise=true , address=145.79.227.8:4335 , piece_id=Q7UEGDD4AGZREL2V2VRHEVDJFIDPKKUF6JCZ2FJGQYJAVW7KQAKQ, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:21.895347667 (580h4m21.766430713s) [noise=true , address=145.79.227.42:4150 , piece_id=TT4PDRJ6HRBNFYKWOBEHOPYSPMYVN2NZZKUBK3E5ACHRKJPK4ZMA, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:21.912551226 (580h4m21.783634261s) [noise=true , address=145.79.227.29:4959 , piece_id=IUPWCJOL4JXCAOUTP4GCQMK657U53VGQD4JXRNNHSYLK2AK56WNA, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:21.918776128 (580h4m21.789859173s) [noise=true , address=145.79.227.6:4283 , piece_id=UZISQ26DIRCHFWIKODJCEQYD2N3FEWFKC7CAL7LRB45KAGPHC5BQ, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.049639643 (580h4m21.920722678s) [noise=true , address=5.231.73.4:4027 , piece_id=W56MVTNRZX4XFEMCJULZEX6GXQ4BNDNJ3R3EZ5LDHWVCGGMB2NTQ, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.062422904 (580h4m21.933505939s) [noise=true , address=77.90.31.17:4786 , piece_id=EJTLQD7JFIXVQ4GZ6S35KOISBSCOVOCW44R7JDDN5HLWRXMZDQ6A, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.087400355 (580h4m21.9584834s) [noise=true , address=77.90.31.18:4836 , piece_id=ONCF7RXVICUL55HJ2AWB466YFRANSOIN4WKMH5VCURXP334CRG5A, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.123477030 (580h4m21.994560075s) [noise=true , address=77.90.31.10:4542 , piece_id=77QBOT4PZXNTMK7Z4SNPYVSJDNKZAJYKZIE2XJO2KOKYNFS6XN6Q, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.147097236 (580h4m22.018180271s) [noise=true , address=77.90.31.10:4539 , piece_id=HFHV4GSLPWU2NXE24SCA7XHPSQMHEKCTT7HOPBXJYQDN5XYQBUTA, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.187114676 (580h4m22.058197721s) [noise=true , address=45.91.116.9:4283 , piece_id=W2IRSXBWJZBSADPBETO254LH5WH7RWHLP65FMYCW3QLTDZMWAJ2A, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.755107038 (580h4m22.626190073s) [noise=true , address=145.79.227.43:4190 , piece_id=5O4HKBMCZZ5FWQCKO2B3NJVBV2ZHAVZCYRYRPHCURU5ZQ2SPIE5A, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.779882074 (580h4m22.650965109s) [noise=true , address=145.79.227.30:4773 , piece_id=BAR4QMZ7OM74IIPLRGWAMEYZKIILXXQNRYJFBUHXV5YUWYD7X2YA, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.820831227 (580h4m22.691914262s) [noise=true , address=145.79.227.44:4213 , piece_id=47WPDBUPSA5QUYOBYSFQFQMIFZMJJTD3LBYL6J4WCM4G2LLYKRBQ, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.820854071 (580h4m22.691937106s) [seg_pos={0 0}, upload=508498]: Segment reached optimal threshold of 14 pieces. Dec 29 19:33:22.821045084 (580h4m22.692128119s) [noise=true , address=145.79.227.34:4913 , piece_id=O22AXRPCITP6BCWH4ZANG2Z2F4PENTRZOCIA3YUMXYBVP7XXRWUA, seg_pos={0 0}, upload=508498]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1228o9xxwySb7RCPH81yobfdRxNQoz82JYgkb9TEw8eeX7CATgC): protocol: expected piece hash; context canceled; EOF Dec 29 19:33:22.821051897 (580h4m22.692134932s) [noise=true , address=145.79.227.34:4913 , piece_id=O22AXRPCITP6BCWH4ZANG2Z2F4PENTRZOCIA3YUMXYBVP7XXRWUA, seg_pos={0 0}, upload=508498]: Long tail cancellation due to optimal threshold Dec 29 19:33:22.821168860 (580h4m22.692251905s) [noise=true , address=64.49.13.169:4711 , piece_id=MZYKDHA7E4KNOZ4C752DWBLJJPJQTBGCRX6QACQCCCIJ5VT4LU4Q, seg_pos={0 0}, upload=508498]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nxT1Ldr46HPrDRHQ2nrwEBZT5bEVoPAAfBz86ZYwUH13sqwHu): protocol: write tcp 188.166.225.170:52944->64.49.13.169:4711: use of closed network connection; write tcp 188.166.225.170:52944->64.49.13.169:4711: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52944->64.49.13.169:4711: use of closed network connection Dec 29 19:33:22.821176013 (580h4m22.692259049s) [noise=true , address=64.49.13.169:4711 , piece_id=MZYKDHA7E4KNOZ4C752DWBLJJPJQTBGCRX6QACQCCCIJ5VT4LU4Q, seg_pos={0 0}, upload=508498]: Long tail cancellation due to optimal threshold Dec 29 19:33:22.821193216 (580h4m22.692276251s) [noise=true , address=64.49.13.27:4966 , piece_id=GABX3GQG7TGY36WBSO3JWMBEIOK7R7MDZA4IHPPNBRBFXBTMJO3A, seg_pos={0 0}, upload=508498]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gUV64TP8cTQhBjswnhcWfQHwYEEWqr5ASuPjtEx3X6ZQNVhS5): protocol: write tcp 188.166.225.170:47300->64.49.13.27:4966: use of closed network connection; write tcp 188.166.225.170:47300->64.49.13.27:4966: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47300->64.49.13.27:4966: use of closed network connection Dec 29 19:33:22.821198947 (580h4m22.692281983s) [noise=true , address=64.49.13.27:4966 , piece_id=GABX3GQG7TGY36WBSO3JWMBEIOK7R7MDZA4IHPPNBRBFXBTMJO3A, seg_pos={0 0}, upload=508498]: Long tail cancellation due to optimal threshold Dec 29 19:33:22.821300189 (580h4m22.692383225s) [noise=true , address=145.79.227.33:4893 , piece_id=QTATPQWRYUQ33FL2M3RAP762AM3AC4PT7BHBCIUT7GFNVDC5UDGA, seg_pos={0 0}, upload=508498]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17VMFZ2T8jrSJVtchciJ9SnyrR6JstvcMoXrN5Q3zTD1aQU1oJ): protocol: expected piece hash; context canceled; EOF Dec 29 19:33:22.821306302 (580h4m22.692389337s) [noise=true , address=145.79.227.33:4893 , piece_id=QTATPQWRYUQ33FL2M3RAP762AM3AC4PT7BHBCIUT7GFNVDC5UDGA, seg_pos={0 0}, upload=508498]: Long tail cancellation due to optimal threshold Dec 29 19:33:22.821399579 (580h4m22.692482614s) [seg_pos={0 0}, upload=508498]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 19:33:22.821465755 (580h4m22.6925488s) [upload=508498]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 19:33:22.821470243 (580h4m22.692553278s) [upload=508498]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 19:33:23.123995567 (580h4m22.995078612s) [upload=508498]: Done uploading segments... Dec 29 19:33:23.124020665 (580h4m22.99510371s) [upload=508498]: Upload finished. err: <nil> Duration: 16.884090089s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing) antiblip: 1 out of 60 is failed (threshold=6) [...............................X............................] duration: 265.352725ms remote: "tcp:207.211.208.129:443" target: https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt trace: https://jaeger.dev.storj.io/trace/4679501fce62ca7 FAIL (2025-12-29 20:17:06 UTC) orbiter warning: Get "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt": context deadline exceeded Duration: 265.395015ms edge linksharing bgp eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [.XXX.XX...X.X....X..X.X....X.......X.....................X..] duration: 5.959096559s trace: https://jaeger.dev.storj.io/trace/607cf963331dd581 FAIL (2025-12-29 20:16:56 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 20:14:59.343887631 (580h45m59.214970666s) [upload=509109]: Done starting upload Dec 29 20:14:59.343929450 (580h45m59.215012485s) [upload=509109]: Uploading segments... Dec 29 20:14:59.344178725 (580h45m59.21526176s) [upload=509109]: Got next segment. Inline: false Dec 29 20:14:59.344213341 (580h45m59.215296376s) [upload=509109]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 20:14:59.344218040 (580h45m59.215301075s) [upload=509109]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 20:14:59.620876250 (580h45m59.491959285s) [seg_pos={0 0}, upload=509109]: Begin upload segment... Dec 29 20:14:59.620932867 (580h45m59.492015912s) [seg_pos={0 0}, upload=509109]: Done begin upload segment. Dec 29 20:14:59.620975569 (580h45m59.492058604s) [noise=true , address=145.79.227.51:4411 , piece_id=LYCY2Z7Z3BFR5XSM6JXFCL5LZAU56PUBVGHLVURZUTQTEZF7CLFQ, seg_pos={0 0}, upload=509109]: Uploading piece... Dec 29 20:14:59.620983133 (580h45m59.492066178s) [noise=true , address=145.79.227.48:4303 , piece_id=UYEJZYH7QYLL6KGKEQBH6DEGEBMWZSM2OT7W4V6D4D3FQV4XUDZQ, seg_pos={0 0}, upload=509109]: Uploading piece... Dec 29 20:14:59.621128750 (580h45m59.492211796s) [noise=true , address=145.79.227.46:4602 , piece_id=ZJA75URLP6YW2ROXGLR4PONX2PXXWZNGGANAI37SREOPJ5S7OV3A, seg_pos={0 0}, upload=509109]: Uploading piece... Dec 29 20:14:59.621771063 (580h45m59.492854098s) [noise=true , address=145.79.227.15:4875 , piece_id=RFIPM3MTYFN5PLV4NVNF7AE5OVA54ZIOR2PUY7S4674CN466ECBA, seg_pos={0 0}, upload=509109]: Uploading piece... Dec 29 20:14:59.622036208 (580h45m59.493119263s) [noise=true , address=145.79.227.18:4006 , piece_id=QP3MYNJN7EGPAM7NB4E2ITEV6EPATVAIWGAPT3KNEUXEPPBNQVDA, seg_pos={0 0}, upload=509109]: Uploading piece... Dec 29 20:14:59.622303207 (580h45m59.493386252s) [noise=true , address=145.79.227.28:4725 , piece_id=U442UUBRULRA3AY6H7EP5G2XJFY47YYDMBWRCMQBM2BLCKZJBYDA, seg_pos={0 0}, upload=509109]: Uploading piece... Dec 29 20:14:59.622693610 (580h45m59.493776645s) [noise=true , address=145.79.227.44:4194 , piece_id=NEQVB6ERVDONQCXWRFXXFPZ3EBSLRJJMWOZHSBCKZWFWZ5FJXTNA, seg_pos={0 0}, upload=509109]: Uploading piece... Dec 29 20:14:59.623383705 (580h45m59.49446674s) [noise=true , address=77.90.31.13:4627 , piece_id=OVJXRA53CFJFZSRJ6H2MHZLO4RCLLC3TIGGYQN3ESNQFA3ESQD3A, seg_pos={0 0}, upload=509109]: Uploading piece... Dec 29 20:14:59.623912161 (580h45m59.494995206s) [noise=true , address=45.91.116.7:4010 , piece_id=BO772OMWJBZDCBAX3O6ECC766FEHZVTOJHKJHDLCSXJWKEKIV6BA, seg_pos={0 0}, upload=509109]: Uploading piece... Dec 29 20:14:59.624389991 (580h45m59.495473026s) [noise=true , address=145.79.227.2:4115 , piece_id=J6ZAJU7FYPAFEI47EJBUUUCASWZY2PZZWABSA7WRLAQPYQMFTENQ, seg_pos={0 0}, upload=509109]: Uploading piece... Dec 29 20:14:59.624497966 (580h45m59.495581001s) [noise=true , address=145.79.227.11:4731 , piece_id=46XM74D3UEXNZXHZYWFOYW6SHHWTQU2HA6Q2WYSJLH7OVC23JALQ, seg_pos={0 0}, upload=509109]: Uploading piece... Dec 29 20:14:59.624935829 (580h45m59.496018864s) [noise=true , address=145.79.227.19:4078 , piece_id=NZLVPFPYJXZHMK2XSIMIDBR7CN4OWVWT5QGNLTK3STC5KBY7CMGA, seg_pos={0 0}, upload=509109]: Uploading piece... Dec 29 20:14:59.624977129 (580h45m59.496060174s) [noise=true , address=45.91.116.9:4262 , piece_id=UV5XMT447DE6QTJYRYBD7466OPDQ3YQSIVQ4C2BSIO37OCZKGSHA, seg_pos={0 0}, upload=509109]: Uploading piece... Dec 29 20:14:59.625338717 (580h45m59.496421762s) [noise=true , address=45.91.116.6:4975 , piece_id=YHNIUJWPFG4EC7FJYL43CCVFTKZ2GMKX5ANKEIHDMLAL2AGA7V3A, seg_pos={0 0}, upload=509109]: Uploading piece... Dec 29 20:14:59.625486658 (580h45m59.496569704s) [noise=true , address=145.79.227.36:4518 , piece_id=3DYS7R2VTEUGCDWBWFP2OZHYTMQGLK7OS3OFSCUBIKWSWHJ44ELQ, seg_pos={0 0}, upload=509109]: Uploading piece... Dec 29 20:14:59.625852676 (580h45m59.496935721s) [noise=true , address=145.79.227.9:4366 , piece_id=ICYZM4QQZQJLS4T4KYM47NEMIKYD7PZWQ2U2NEFLAWLLPO6OFG3A, seg_pos={0 0}, upload=509109]: Uploading piece... Dec 29 20:15:01.223830613 (580h46m1.094913658s) [upload=509109]: Next returned nil segment Dec 29 20:15:01.223865600 (580h46m1.094948645s) [upload=509109]: Waiting for error group managing segments... Dec 29 20:15:01.599999537 (580h46m1.471082582s) [noise=true , address=145.79.227.11:4731 , piece_id=46XM74D3UEXNZXHZYWFOYW6SHHWTQU2HA6Q2WYSJLH7OVC23JALQ, seg_pos={0 0}, upload=509109]: Done uploading piece. err: <nil> Dec 29 20:15:01.606734191 (580h46m1.477817236s) [noise=true , address=145.79.227.9:4366 , piece_id=ICYZM4QQZQJLS4T4KYM47NEMIKYD7PZWQ2U2NEFLAWLLPO6OFG3A, seg_pos={0 0}, upload=509109]: Done uploading piece. err: <nil> Dec 29 20:15:01.609790219 (580h46m1.480873265s) [noise=true , address=145.79.227.48:4303 , piece_id=UYEJZYH7QYLL6KGKEQBH6DEGEBMWZSM2OT7W4V6D4D3FQV4XUDZQ, seg_pos={0 0}, upload=509109]: Done uploading piece. err: <nil> Dec 29 20:15:01.613856723 (580h46m1.484939768s) [noise=true , address=145.79.227.51:4411 , piece_id=LYCY2Z7Z3BFR5XSM6JXFCL5LZAU56PUBVGHLVURZUTQTEZF7CLFQ, seg_pos={0 0}, upload=509109]: Done uploading piece. err: <nil> Dec 29 20:15:01.615434938 (580h46m1.486517973s) [noise=true , address=145.79.227.46:4602 , piece_id=ZJA75URLP6YW2ROXGLR4PONX2PXXWZNGGANAI37SREOPJ5S7OV3A, seg_pos={0 0}, upload=509109]: Done uploading piece. err: <nil> Dec 29 20:15:01.617581145 (580h46m1.48866419s) [noise=true , address=145.79.227.19:4078 , piece_id=NZLVPFPYJXZHMK2XSIMIDBR7CN4OWVWT5QGNLTK3STC5KBY7CMGA, seg_pos={0 0}, upload=509109]: Done uploading piece. err: <nil> Dec 29 20:15:01.618039818 (580h46m1.489122863s) [noise=true , address=145.79.227.15:4875 , piece_id=RFIPM3MTYFN5PLV4NVNF7AE5OVA54ZIOR2PUY7S4674CN466ECBA, seg_pos={0 0}, upload=509109]: Done uploading piece. err: <nil> Dec 29 20:15:01.624408906 (580h46m1.495491951s) [noise=true , address=145.79.227.36:4518 , piece_id=3DYS7R2VTEUGCDWBWFP2OZHYTMQGLK7OS3OFSCUBIKWSWHJ44ELQ, seg_pos={0 0}, upload=509109]: Done uploading piece. err: <nil> Dec 29 20:15:01.629708417 (580h46m1.500791482s) [noise=true , address=145.79.227.2:4115 , piece_id=J6ZAJU7FYPAFEI47EJBUUUCASWZY2PZZWABSA7WRLAQPYQMFTENQ, seg_pos={0 0}, upload=509109]: Done uploading piece. err: <nil> Dec 29 20:15:01.654622637 (580h46m1.525705682s) [noise=true , address=145.79.227.18:4006 , piece_id=QP3MYNJN7EGPAM7NB4E2ITEV6EPATVAIWGAPT3KNEUXEPPBNQVDA, seg_pos={0 0}, upload=509109]: Done uploading piece. err: <nil> Dec 29 20:15:01.799321647 (580h46m1.670404682s) [noise=true , address=77.90.31.13:4627 , piece_id=OVJXRA53CFJFZSRJ6H2MHZLO4RCLLC3TIGGYQN3ESNQFA3ESQD3A, seg_pos={0 0}, upload=509109]: Done uploading piece. err: <nil> Dec 29 20:15:01.902753888 (580h46m1.773836943s) [noise=true , address=45.91.116.7:4010 , piece_id=BO772OMWJBZDCBAX3O6ECC766FEHZVTOJHKJHDLCSXJWKEKIV6BA, seg_pos={0 0}, upload=509109]: Done uploading piece. err: <nil> Dec 29 20:15:02.167078173 (580h46m2.038161228s) [noise=true , address=45.91.116.9:4262 , piece_id=UV5XMT447DE6QTJYRYBD7466OPDQ3YQSIVQ4C2BSIO37OCZKGSHA, seg_pos={0 0}, upload=509109]: Done uploading piece. err: <nil> Dec 29 20:15:02.167162874 (580h46m2.038245929s) [seg_pos={0 0}, upload=509109]: Segment reached optimal threshold of 13 pieces. Dec 29 20:15:02.170200207 (580h46m2.041283262s) [noise=true , address=45.91.116.6:4975 , piece_id=YHNIUJWPFG4EC7FJYL43CCVFTKZ2GMKX5ANKEIHDMLAL2AGA7V3A, seg_pos={0 0}, upload=509109]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: read tcp 188.166.225.170:42964->45.91.116.6:4975: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 29 20:15:02.170215235 (580h46m2.04129829s) [noise=true , address=45.91.116.6:4975 , piece_id=YHNIUJWPFG4EC7FJYL43CCVFTKZ2GMKX5ANKEIHDMLAL2AGA7V3A, seg_pos={0 0}, upload=509109]: Long tail cancellation due to optimal threshold Dec 29 20:15:02.170672046 (580h46m2.041755101s) [noise=true , address=145.79.227.44:4194 , piece_id=NEQVB6ERVDONQCXWRFXXFPZ3EBSLRJJMWOZHSBCKZWFWZ5FJXTNA, seg_pos={0 0}, upload=509109]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh): protocol: expected piece hash; context canceled; EOF Dec 29 20:15:02.170683507 (580h46m2.041766553s) [noise=true , address=145.79.227.44:4194 , piece_id=NEQVB6ERVDONQCXWRFXXFPZ3EBSLRJJMWOZHSBCKZWFWZ5FJXTNA, seg_pos={0 0}, upload=509109]: Long tail cancellation due to optimal threshold Dec 29 20:15:02.170846298 (580h46m2.041929353s) [noise=true , address=145.79.227.28:4725 , piece_id=U442UUBRULRA3AY6H7EP5G2XJFY47YYDMBWRCMQBM2BLCKZJBYDA, seg_pos={0 0}, upload=509109]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12YKLTah3swX14kL82eNogJxd8D8Hjfq7SPjDDQAapDnXmDctP1): protocol: write tcp 188.166.225.170:56476->145.79.227.28:4725: use of closed network connection; write tcp 188.166.225.170:56476->145.79.227.28:4725: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56476->145.79.227.28:4725: use of closed network connection Dec 29 20:15:02.170859112 (580h46m2.041942157s) [noise=true , address=145.79.227.28:4725 , piece_id=U442UUBRULRA3AY6H7EP5G2XJFY47YYDMBWRCMQBM2BLCKZJBYDA, seg_pos={0 0}, upload=509109]: Long tail cancellation due to optimal threshold Dec 29 20:15:02.170884811 (580h46m2.041967856s) [seg_pos={0 0}, upload=509109]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 20:15:02.171053743 (580h46m2.042136788s) [upload=509109]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 20:15:02.171062129 (580h46m2.042145184s) [upload=509109]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 20:15:02.455142652 (580h46m2.326225707s) [upload=509109]: Done uploading segments... Dec 29 20:15:02.455171597 (580h46m2.326254642s) [upload=509109]: Upload finished. err: <nil> Duration: 5.959127787s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 15.409983061s trace: https://jaeger.dev.storj.io/trace/24ac2b258db6f49 FAIL (2025-12-29 20:16:39 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 19.881883s) [noise=true , address=145.79.227.44:4213 , piece_id=47WPDBUPSA5QUYOBYSFQFQMIFZMJJTD3LBYL6J4WCM4G2LLYKRBQ, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.011913035 (580h4m19.88299608s) [noise=true , address=145.79.227.11:4748 , piece_id=3FFT5U7HDYRGKI67WTT3EWL6PSQY4MOFM2RVALUI2SXDQVFAUV7A, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.012333796 (580h4m19.883416831s) [noise=true , address=45.91.116.9:4283 , piece_id=W2IRSXBWJZBSADPBETO254LH5WH7RWHLP65FMYCW3QLTDZMWAJ2A, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.012776579 (580h4m19.883859614s) [noise=true , address=145.79.227.6:4283 , piece_id=UZISQ26DIRCHFWIKODJCEQYD2N3FEWFKC7CAL7LRB45KAGPHC5BQ, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.013135943 (580h4m19.884218978s) [noise=true , address=145.79.227.42:4150 , piece_id=TT4PDRJ6HRBNFYKWOBEHOPYSPMYVN2NZZKUBK3E5ACHRKJPK4ZMA, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.013460732 (580h4m19.884543767s) [noise=true , address=77.90.31.17:4786 , piece_id=EJTLQD7JFIXVQ4GZ6S35KOISBSCOVOCW44R7JDDN5HLWRXMZDQ6A, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.013853229 (580h4m19.884936264s) [noise=true , address=145.79.227.29:4959 , piece_id=IUPWCJOL4JXCAOUTP4GCQMK657U53VGQD4JXRNNHSYLK2AK56WNA, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.014144343 (580h4m19.885227388s) [noise=true , address=145.79.227.33:4893 , piece_id=QTATPQWRYUQ33FL2M3RAP762AM3AC4PT7BHBCIUT7GFNVDC5UDGA, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.014250846 (580h4m19.885333881s) [noise=true , address=5.231.73.4:4027 , piece_id=W56MVTNRZX4XFEMCJULZEX6GXQ4BNDNJ3R3EZ5LDHWVCGGMB2NTQ, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.014906585 (580h4m19.88598962s) [noise=true , address=145.79.227.30:4773 , piece_id=BAR4QMZ7OM74IIPLRGWAMEYZKIILXXQNRYJFBUHXV5YUWYD7X2YA, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.015511426 (580h4m19.886594471s) [noise=true , address=64.49.13.169:4711 , piece_id=MZYKDHA7E4KNOZ4C752DWBLJJPJQTBGCRX6QACQCCCIJ5VT4LU4Q, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.016659873 (580h4m19.887742918s) [noise=true , address=145.79.227.8:4335 , piece_id=Q7UEGDD4AGZREL2V2VRHEVDJFIDPKKUF6JCZ2FJGQYJAVW7KQAKQ, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.018312429 (580h4m19.889395464s) [noise=true , address=64.49.13.27:4966 , piece_id=GABX3GQG7TGY36WBSO3JWMBEIOK7R7MDZA4IHPPNBRBFXBTMJO3A, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.019851220 (580h4m19.890934255s) [noise=true , address=77.90.31.18:4836 , piece_id=ONCF7RXVICUL55HJ2AWB466YFRANSOIN4WKMH5VCURXP334CRG5A, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.021306711 (580h4m19.892389746s) [noise=true , address=145.79.227.34:4913 , piece_id=O22AXRPCITP6BCWH4ZANG2Z2F4PENTRZOCIA3YUMXYBVP7XXRWUA, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:21.467852993 (580h4m21.338936028s) [upload=508498]: Next returned nil segment Dec 29 19:33:21.467865737 (580h4m21.338948772s) [upload=508498]: Waiting for error group managing segments... Dec 29 19:33:21.864176322 (580h4m21.735259357s) [noise=true , address=145.79.227.11:4748 , piece_id=3FFT5U7HDYRGKI67WTT3EWL6PSQY4MOFM2RVALUI2SXDQVFAUV7A, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:21.890454300 (580h4m21.761537345s) [noise=true , address=145.79.227.8:4335 , piece_id=Q7UEGDD4AGZREL2V2VRHEVDJFIDPKKUF6JCZ2FJGQYJAVW7KQAKQ, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:21.895347667 (580h4m21.766430713s) [noise=true , address=145.79.227.42:4150 , piece_id=TT4PDRJ6HRBNFYKWOBEHOPYSPMYVN2NZZKUBK3E5ACHRKJPK4ZMA, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:21.912551226 (580h4m21.783634261s) [noise=true , address=145.79.227.29:4959 , piece_id=IUPWCJOL4JXCAOUTP4GCQMK657U53VGQD4JXRNNHSYLK2AK56WNA, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:21.918776128 (580h4m21.789859173s) [noise=true , address=145.79.227.6:4283 , piece_id=UZISQ26DIRCHFWIKODJCEQYD2N3FEWFKC7CAL7LRB45KAGPHC5BQ, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.049639643 (580h4m21.920722678s) [noise=true , address=5.231.73.4:4027 , piece_id=W56MVTNRZX4XFEMCJULZEX6GXQ4BNDNJ3R3EZ5LDHWVCGGMB2NTQ, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.062422904 (580h4m21.933505939s) [noise=true , address=77.90.31.17:4786 , piece_id=EJTLQD7JFIXVQ4GZ6S35KOISBSCOVOCW44R7JDDN5HLWRXMZDQ6A, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.087400355 (580h4m21.9584834s) [noise=true , address=77.90.31.18:4836 , piece_id=ONCF7RXVICUL55HJ2AWB466YFRANSOIN4WKMH5VCURXP334CRG5A, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.123477030 (580h4m21.994560075s) [noise=true , address=77.90.31.10:4542 , piece_id=77QBOT4PZXNTMK7Z4SNPYVSJDNKZAJYKZIE2XJO2KOKYNFS6XN6Q, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.147097236 (580h4m22.018180271s) [noise=true , address=77.90.31.10:4539 , piece_id=HFHV4GSLPWU2NXE24SCA7XHPSQMHEKCTT7HOPBXJYQDN5XYQBUTA, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.187114676 (580h4m22.058197721s) [noise=true , address=45.91.116.9:4283 , piece_id=W2IRSXBWJZBSADPBETO254LH5WH7RWHLP65FMYCW3QLTDZMWAJ2A, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.755107038 (580h4m22.626190073s) [noise=true , address=145.79.227.43:4190 , piece_id=5O4HKBMCZZ5FWQCKO2B3NJVBV2ZHAVZCYRYRPHCURU5ZQ2SPIE5A, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.779882074 (580h4m22.650965109s) [noise=true , address=145.79.227.30:4773 , piece_id=BAR4QMZ7OM74IIPLRGWAMEYZKIILXXQNRYJFBUHXV5YUWYD7X2YA, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.820831227 (580h4m22.691914262s) [noise=true , address=145.79.227.44:4213 , piece_id=47WPDBUPSA5QUYOBYSFQFQMIFZMJJTD3LBYL6J4WCM4G2LLYKRBQ, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.820854071 (580h4m22.691937106s) [seg_pos={0 0}, upload=508498]: Segment reached optimal threshold of 14 pieces. Dec 29 19:33:22.821045084 (580h4m22.692128119s) [noise=true , address=145.79.227.34:4913 , piece_id=O22AXRPCITP6BCWH4ZANG2Z2F4PENTRZOCIA3YUMXYBVP7XXRWUA, seg_pos={0 0}, upload=508498]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1228o9xxwySb7RCPH81yobfdRxNQoz82JYgkb9TEw8eeX7CATgC): protocol: expected piece hash; context canceled; EOF Dec 29 19:33:22.821051897 (580h4m22.692134932s) [noise=true , address=145.79.227.34:4913 , piece_id=O22AXRPCITP6BCWH4ZANG2Z2F4PENTRZOCIA3YUMXYBVP7XXRWUA, seg_pos={0 0}, upload=508498]: Long tail cancellation due to optimal threshold Dec 29 19:33:22.821168860 (580h4m22.692251905s) [noise=true , address=64.49.13.169:4711 , piece_id=MZYKDHA7E4KNOZ4C752DWBLJJPJQTBGCRX6QACQCCCIJ5VT4LU4Q, seg_pos={0 0}, upload=508498]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nxT1Ldr46HPrDRHQ2nrwEBZT5bEVoPAAfBz86ZYwUH13sqwHu): protocol: write tcp 188.166.225.170:52944->64.49.13.169:4711: use of closed network connection; write tcp 188.166.225.170:52944->64.49.13.169:4711: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52944->64.49.13.169:4711: use of closed network connection Dec 29 19:33:22.821176013 (580h4m22.692259049s) [noise=true , address=64.49.13.169:4711 , piece_id=MZYKDHA7E4KNOZ4C752DWBLJJPJQTBGCRX6QACQCCCIJ5VT4LU4Q, seg_pos={0 0}, upload=508498]: Long tail cancellation due to optimal threshold Dec 29 19:33:22.821193216 (580h4m22.692276251s) [noise=true , address=64.49.13.27:4966 , piece_id=GABX3GQG7TGY36WBSO3JWMBEIOK7R7MDZA4IHPPNBRBFXBTMJO3A, seg_pos={0 0}, upload=508498]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gUV64TP8cTQhBjswnhcWfQHwYEEWqr5ASuPjtEx3X6ZQNVhS5): protocol: write tcp 188.166.225.170:47300->64.49.13.27:4966: use of closed network connection; write tcp 188.166.225.170:47300->64.49.13.27:4966: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47300->64.49.13.27:4966: use of closed network connection Dec 29 19:33:22.821198947 (580h4m22.692281983s) [noise=true , address=64.49.13.27:4966 , piece_id=GABX3GQG7TGY36WBSO3JWMBEIOK7R7MDZA4IHPPNBRBFXBTMJO3A, seg_pos={0 0}, upload=508498]: Long tail cancellation due to optimal threshold Dec 29 19:33:22.821300189 (580h4m22.692383225s) [noise=true , address=145.79.227.33:4893 , piece_id=QTATPQWRYUQ33FL2M3RAP762AM3AC4PT7BHBCIUT7GFNVDC5UDGA, seg_pos={0 0}, upload=508498]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17VMFZ2T8jrSJVtchciJ9SnyrR6JstvcMoXrN5Q3zTD1aQU1oJ): protocol: expected piece hash; context canceled; EOF Dec 29 19:33:22.821306302 (580h4m22.692389337s) [noise=true , address=145.79.227.33:4893 , piece_id=QTATPQWRYUQ33FL2M3RAP762AM3AC4PT7BHBCIUT7GFNVDC5UDGA, seg_pos={0 0}, upload=508498]: Long tail cancellation due to optimal threshold Dec 29 19:33:22.821399579 (580h4m22.692482614s) [seg_pos={0 0}, upload=508498]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 19:33:22.821465755 (580h4m22.6925488s) [upload=508498]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 19:33:22.821470243 (580h4m22.692553278s) [upload=508498]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 19:33:23.123995567 (580h4m22.995078612s) [upload=508498]: Done uploading segments... Dec 29 19:33:23.124020665 (580h4m22.99510371s) [upload=508498]: Upload finished. err: <nil> Duration: 15.410007838s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 18.243300432s trace: https://jaeger.dev.storj.io/trace/24ac2b258db6f49 FAIL (2025-12-29 20:16:15 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 19.881883s) [noise=true , address=145.79.227.44:4213 , piece_id=47WPDBUPSA5QUYOBYSFQFQMIFZMJJTD3LBYL6J4WCM4G2LLYKRBQ, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.011913035 (580h4m19.88299608s) [noise=true , address=145.79.227.11:4748 , piece_id=3FFT5U7HDYRGKI67WTT3EWL6PSQY4MOFM2RVALUI2SXDQVFAUV7A, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.012333796 (580h4m19.883416831s) [noise=true , address=45.91.116.9:4283 , piece_id=W2IRSXBWJZBSADPBETO254LH5WH7RWHLP65FMYCW3QLTDZMWAJ2A, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.012776579 (580h4m19.883859614s) [noise=true , address=145.79.227.6:4283 , piece_id=UZISQ26DIRCHFWIKODJCEQYD2N3FEWFKC7CAL7LRB45KAGPHC5BQ, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.013135943 (580h4m19.884218978s) [noise=true , address=145.79.227.42:4150 , piece_id=TT4PDRJ6HRBNFYKWOBEHOPYSPMYVN2NZZKUBK3E5ACHRKJPK4ZMA, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.013460732 (580h4m19.884543767s) [noise=true , address=77.90.31.17:4786 , piece_id=EJTLQD7JFIXVQ4GZ6S35KOISBSCOVOCW44R7JDDN5HLWRXMZDQ6A, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.013853229 (580h4m19.884936264s) [noise=true , address=145.79.227.29:4959 , piece_id=IUPWCJOL4JXCAOUTP4GCQMK657U53VGQD4JXRNNHSYLK2AK56WNA, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.014144343 (580h4m19.885227388s) [noise=true , address=145.79.227.33:4893 , piece_id=QTATPQWRYUQ33FL2M3RAP762AM3AC4PT7BHBCIUT7GFNVDC5UDGA, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.014250846 (580h4m19.885333881s) [noise=true , address=5.231.73.4:4027 , piece_id=W56MVTNRZX4XFEMCJULZEX6GXQ4BNDNJ3R3EZ5LDHWVCGGMB2NTQ, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.014906585 (580h4m19.88598962s) [noise=true , address=145.79.227.30:4773 , piece_id=BAR4QMZ7OM74IIPLRGWAMEYZKIILXXQNRYJFBUHXV5YUWYD7X2YA, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.015511426 (580h4m19.886594471s) [noise=true , address=64.49.13.169:4711 , piece_id=MZYKDHA7E4KNOZ4C752DWBLJJPJQTBGCRX6QACQCCCIJ5VT4LU4Q, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.016659873 (580h4m19.887742918s) [noise=true , address=145.79.227.8:4335 , piece_id=Q7UEGDD4AGZREL2V2VRHEVDJFIDPKKUF6JCZ2FJGQYJAVW7KQAKQ, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.018312429 (580h4m19.889395464s) [noise=true , address=64.49.13.27:4966 , piece_id=GABX3GQG7TGY36WBSO3JWMBEIOK7R7MDZA4IHPPNBRBFXBTMJO3A, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.019851220 (580h4m19.890934255s) [noise=true , address=77.90.31.18:4836 , piece_id=ONCF7RXVICUL55HJ2AWB466YFRANSOIN4WKMH5VCURXP334CRG5A, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.021306711 (580h4m19.892389746s) [noise=true , address=145.79.227.34:4913 , piece_id=O22AXRPCITP6BCWH4ZANG2Z2F4PENTRZOCIA3YUMXYBVP7XXRWUA, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:21.467852993 (580h4m21.338936028s) [upload=508498]: Next returned nil segment Dec 29 19:33:21.467865737 (580h4m21.338948772s) [upload=508498]: Waiting for error group managing segments... Dec 29 19:33:21.864176322 (580h4m21.735259357s) [noise=true , address=145.79.227.11:4748 , piece_id=3FFT5U7HDYRGKI67WTT3EWL6PSQY4MOFM2RVALUI2SXDQVFAUV7A, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:21.890454300 (580h4m21.761537345s) [noise=true , address=145.79.227.8:4335 , piece_id=Q7UEGDD4AGZREL2V2VRHEVDJFIDPKKUF6JCZ2FJGQYJAVW7KQAKQ, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:21.895347667 (580h4m21.766430713s) [noise=true , address=145.79.227.42:4150 , piece_id=TT4PDRJ6HRBNFYKWOBEHOPYSPMYVN2NZZKUBK3E5ACHRKJPK4ZMA, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:21.912551226 (580h4m21.783634261s) [noise=true , address=145.79.227.29:4959 , piece_id=IUPWCJOL4JXCAOUTP4GCQMK657U53VGQD4JXRNNHSYLK2AK56WNA, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:21.918776128 (580h4m21.789859173s) [noise=true , address=145.79.227.6:4283 , piece_id=UZISQ26DIRCHFWIKODJCEQYD2N3FEWFKC7CAL7LRB45KAGPHC5BQ, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.049639643 (580h4m21.920722678s) [noise=true , address=5.231.73.4:4027 , piece_id=W56MVTNRZX4XFEMCJULZEX6GXQ4BNDNJ3R3EZ5LDHWVCGGMB2NTQ, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.062422904 (580h4m21.933505939s) [noise=true , address=77.90.31.17:4786 , piece_id=EJTLQD7JFIXVQ4GZ6S35KOISBSCOVOCW44R7JDDN5HLWRXMZDQ6A, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.087400355 (580h4m21.9584834s) [noise=true , address=77.90.31.18:4836 , piece_id=ONCF7RXVICUL55HJ2AWB466YFRANSOIN4WKMH5VCURXP334CRG5A, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.123477030 (580h4m21.994560075s) [noise=true , address=77.90.31.10:4542 , piece_id=77QBOT4PZXNTMK7Z4SNPYVSJDNKZAJYKZIE2XJO2KOKYNFS6XN6Q, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.147097236 (580h4m22.018180271s) [noise=true , address=77.90.31.10:4539 , piece_id=HFHV4GSLPWU2NXE24SCA7XHPSQMHEKCTT7HOPBXJYQDN5XYQBUTA, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.187114676 (580h4m22.058197721s) [noise=true , address=45.91.116.9:4283 , piece_id=W2IRSXBWJZBSADPBETO254LH5WH7RWHLP65FMYCW3QLTDZMWAJ2A, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.755107038 (580h4m22.626190073s) [noise=true , address=145.79.227.43:4190 , piece_id=5O4HKBMCZZ5FWQCKO2B3NJVBV2ZHAVZCYRYRPHCURU5ZQ2SPIE5A, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.779882074 (580h4m22.650965109s) [noise=true , address=145.79.227.30:4773 , piece_id=BAR4QMZ7OM74IIPLRGWAMEYZKIILXXQNRYJFBUHXV5YUWYD7X2YA, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.820831227 (580h4m22.691914262s) [noise=true , address=145.79.227.44:4213 , piece_id=47WPDBUPSA5QUYOBYSFQFQMIFZMJJTD3LBYL6J4WCM4G2LLYKRBQ, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.820854071 (580h4m22.691937106s) [seg_pos={0 0}, upload=508498]: Segment reached optimal threshold of 14 pieces. Dec 29 19:33:22.821045084 (580h4m22.692128119s) [noise=true , address=145.79.227.34:4913 , piece_id=O22AXRPCITP6BCWH4ZANG2Z2F4PENTRZOCIA3YUMXYBVP7XXRWUA, seg_pos={0 0}, upload=508498]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1228o9xxwySb7RCPH81yobfdRxNQoz82JYgkb9TEw8eeX7CATgC): protocol: expected piece hash; context canceled; EOF Dec 29 19:33:22.821051897 (580h4m22.692134932s) [noise=true , address=145.79.227.34:4913 , piece_id=O22AXRPCITP6BCWH4ZANG2Z2F4PENTRZOCIA3YUMXYBVP7XXRWUA, seg_pos={0 0}, upload=508498]: Long tail cancellation due to optimal threshold Dec 29 19:33:22.821168860 (580h4m22.692251905s) [noise=true , address=64.49.13.169:4711 , piece_id=MZYKDHA7E4KNOZ4C752DWBLJJPJQTBGCRX6QACQCCCIJ5VT4LU4Q, seg_pos={0 0}, upload=508498]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nxT1Ldr46HPrDRHQ2nrwEBZT5bEVoPAAfBz86ZYwUH13sqwHu): protocol: write tcp 188.166.225.170:52944->64.49.13.169:4711: use of closed network connection; write tcp 188.166.225.170:52944->64.49.13.169:4711: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52944->64.49.13.169:4711: use of closed network connection Dec 29 19:33:22.821176013 (580h4m22.692259049s) [noise=true , address=64.49.13.169:4711 , piece_id=MZYKDHA7E4KNOZ4C752DWBLJJPJQTBGCRX6QACQCCCIJ5VT4LU4Q, seg_pos={0 0}, upload=508498]: Long tail cancellation due to optimal threshold Dec 29 19:33:22.821193216 (580h4m22.692276251s) [noise=true , address=64.49.13.27:4966 , piece_id=GABX3GQG7TGY36WBSO3JWMBEIOK7R7MDZA4IHPPNBRBFXBTMJO3A, seg_pos={0 0}, upload=508498]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gUV64TP8cTQhBjswnhcWfQHwYEEWqr5ASuPjtEx3X6ZQNVhS5): protocol: write tcp 188.166.225.170:47300->64.49.13.27:4966: use of closed network connection; write tcp 188.166.225.170:47300->64.49.13.27:4966: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47300->64.49.13.27:4966: use of closed network connection Dec 29 19:33:22.821198947 (580h4m22.692281983s) [noise=true , address=64.49.13.27:4966 , piece_id=GABX3GQG7TGY36WBSO3JWMBEIOK7R7MDZA4IHPPNBRBFXBTMJO3A, seg_pos={0 0}, upload=508498]: Long tail cancellation due to optimal threshold Dec 29 19:33:22.821300189 (580h4m22.692383225s) [noise=true , address=145.79.227.33:4893 , piece_id=QTATPQWRYUQ33FL2M3RAP762AM3AC4PT7BHBCIUT7GFNVDC5UDGA, seg_pos={0 0}, upload=508498]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17VMFZ2T8jrSJVtchciJ9SnyrR6JstvcMoXrN5Q3zTD1aQU1oJ): protocol: expected piece hash; context canceled; EOF Dec 29 19:33:22.821306302 (580h4m22.692389337s) [noise=true , address=145.79.227.33:4893 , piece_id=QTATPQWRYUQ33FL2M3RAP762AM3AC4PT7BHBCIUT7GFNVDC5UDGA, seg_pos={0 0}, upload=508498]: Long tail cancellation due to optimal threshold Dec 29 19:33:22.821399579 (580h4m22.692482614s) [seg_pos={0 0}, upload=508498]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 19:33:22.821465755 (580h4m22.6925488s) [upload=508498]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 19:33:22.821470243 (580h4m22.692553278s) [upload=508498]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 19:33:23.123995567 (580h4m22.995078612s) [upload=508498]: Done uploading segments... Dec 29 19:33:23.124020665 (580h4m22.99510371s) [upload=508498]: Upload finished. err: <nil> Duration: 18.243328375s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing) antiblip: 1 out of 60 is failed (threshold=6) [................................X...........................] duration: 245.48481ms remote: "tcp:207.211.208.129:443" target: https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt trace: https://jaeger.dev.storj.io/trace/4679501fce62ca7 FAIL (2025-12-29 20:16:12 UTC) orbiter warning: Get "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt": context deadline exceeded Duration: 245.509597ms edge linksharing bgp eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [..XXX.XX...X.X....X..X.X....X.......X.....................X.] duration: 6.09757964s trace: https://jaeger.dev.storj.io/trace/607cf963331dd581 FAIL (2025-12-29 20:15:54 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 20:14:59.343887631 (580h45m59.214970666s) [upload=509109]: Done starting upload Dec 29 20:14:59.343929450 (580h45m59.215012485s) [upload=509109]: Uploading segments... Dec 29 20:14:59.344178725 (580h45m59.21526176s) [upload=509109]: Got next segment. Inline: false Dec 29 20:14:59.344213341 (580h45m59.215296376s) [upload=509109]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 20:14:59.344218040 (580h45m59.215301075s) [upload=509109]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 20:14:59.620876250 (580h45m59.491959285s) [seg_pos={0 0}, upload=509109]: Begin upload segment... Dec 29 20:14:59.620932867 (580h45m59.492015912s) [seg_pos={0 0}, upload=509109]: Done begin upload segment. Dec 29 20:14:59.620975569 (580h45m59.492058604s) [noise=true , address=145.79.227.51:4411 , piece_id=LYCY2Z7Z3BFR5XSM6JXFCL5LZAU56PUBVGHLVURZUTQTEZF7CLFQ, seg_pos={0 0}, upload=509109]: Uploading piece... Dec 29 20:14:59.620983133 (580h45m59.492066178s) [noise=true , address=145.79.227.48:4303 , piece_id=UYEJZYH7QYLL6KGKEQBH6DEGEBMWZSM2OT7W4V6D4D3FQV4XUDZQ, seg_pos={0 0}, upload=509109]: Uploading piece... Dec 29 20:14:59.621128750 (580h45m59.492211796s) [noise=true , address=145.79.227.46:4602 , piece_id=ZJA75URLP6YW2ROXGLR4PONX2PXXWZNGGANAI37SREOPJ5S7OV3A, seg_pos={0 0}, upload=509109]: Uploading piece... Dec 29 20:14:59.621771063 (580h45m59.492854098s) [noise=true , address=145.79.227.15:4875 , piece_id=RFIPM3MTYFN5PLV4NVNF7AE5OVA54ZIOR2PUY7S4674CN466ECBA, seg_pos={0 0}, upload=509109]: Uploading piece... Dec 29 20:14:59.622036208 (580h45m59.493119263s) [noise=true , address=145.79.227.18:4006 , piece_id=QP3MYNJN7EGPAM7NB4E2ITEV6EPATVAIWGAPT3KNEUXEPPBNQVDA, seg_pos={0 0}, upload=509109]: Uploading piece... Dec 29 20:14:59.622303207 (580h45m59.493386252s) [noise=true , address=145.79.227.28:4725 , piece_id=U442UUBRULRA3AY6H7EP5G2XJFY47YYDMBWRCMQBM2BLCKZJBYDA, seg_pos={0 0}, upload=509109]: Uploading piece... Dec 29 20:14:59.622693610 (580h45m59.493776645s) [noise=true , address=145.79.227.44:4194 , piece_id=NEQVB6ERVDONQCXWRFXXFPZ3EBSLRJJMWOZHSBCKZWFWZ5FJXTNA, seg_pos={0 0}, upload=509109]: Uploading piece... Dec 29 20:14:59.623383705 (580h45m59.49446674s) [noise=true , address=77.90.31.13:4627 , piece_id=OVJXRA53CFJFZSRJ6H2MHZLO4RCLLC3TIGGYQN3ESNQFA3ESQD3A, seg_pos={0 0}, upload=509109]: Uploading piece... Dec 29 20:14:59.623912161 (580h45m59.494995206s) [noise=true , address=45.91.116.7:4010 , piece_id=BO772OMWJBZDCBAX3O6ECC766FEHZVTOJHKJHDLCSXJWKEKIV6BA, seg_pos={0 0}, upload=509109]: Uploading piece... Dec 29 20:14:59.624389991 (580h45m59.495473026s) [noise=true , address=145.79.227.2:4115 , piece_id=J6ZAJU7FYPAFEI47EJBUUUCASWZY2PZZWABSA7WRLAQPYQMFTENQ, seg_pos={0 0}, upload=509109]: Uploading piece... Dec 29 20:14:59.624497966 (580h45m59.495581001s) [noise=true , address=145.79.227.11:4731 , piece_id=46XM74D3UEXNZXHZYWFOYW6SHHWTQU2HA6Q2WYSJLH7OVC23JALQ, seg_pos={0 0}, upload=509109]: Uploading piece... Dec 29 20:14:59.624935829 (580h45m59.496018864s) [noise=true , address=145.79.227.19:4078 , piece_id=NZLVPFPYJXZHMK2XSIMIDBR7CN4OWVWT5QGNLTK3STC5KBY7CMGA, seg_pos={0 0}, upload=509109]: Uploading piece... Dec 29 20:14:59.624977129 (580h45m59.496060174s) [noise=true , address=45.91.116.9:4262 , piece_id=UV5XMT447DE6QTJYRYBD7466OPDQ3YQSIVQ4C2BSIO37OCZKGSHA, seg_pos={0 0}, upload=509109]: Uploading piece... Dec 29 20:14:59.625338717 (580h45m59.496421762s) [noise=true , address=45.91.116.6:4975 , piece_id=YHNIUJWPFG4EC7FJYL43CCVFTKZ2GMKX5ANKEIHDMLAL2AGA7V3A, seg_pos={0 0}, upload=509109]: Uploading piece... Dec 29 20:14:59.625486658 (580h45m59.496569704s) [noise=true , address=145.79.227.36:4518 , piece_id=3DYS7R2VTEUGCDWBWFP2OZHYTMQGLK7OS3OFSCUBIKWSWHJ44ELQ, seg_pos={0 0}, upload=509109]: Uploading piece... Dec 29 20:14:59.625852676 (580h45m59.496935721s) [noise=true , address=145.79.227.9:4366 , piece_id=ICYZM4QQZQJLS4T4KYM47NEMIKYD7PZWQ2U2NEFLAWLLPO6OFG3A, seg_pos={0 0}, upload=509109]: Uploading piece... Dec 29 20:15:01.223830613 (580h46m1.094913658s) [upload=509109]: Next returned nil segment Dec 29 20:15:01.223865600 (580h46m1.094948645s) [upload=509109]: Waiting for error group managing segments... Dec 29 20:15:01.599999537 (580h46m1.471082582s) [noise=true , address=145.79.227.11:4731 , piece_id=46XM74D3UEXNZXHZYWFOYW6SHHWTQU2HA6Q2WYSJLH7OVC23JALQ, seg_pos={0 0}, upload=509109]: Done uploading piece. err: <nil> Dec 29 20:15:01.606734191 (580h46m1.477817236s) [noise=true , address=145.79.227.9:4366 , piece_id=ICYZM4QQZQJLS4T4KYM47NEMIKYD7PZWQ2U2NEFLAWLLPO6OFG3A, seg_pos={0 0}, upload=509109]: Done uploading piece. err: <nil> Dec 29 20:15:01.609790219 (580h46m1.480873265s) [noise=true , address=145.79.227.48:4303 , piece_id=UYEJZYH7QYLL6KGKEQBH6DEGEBMWZSM2OT7W4V6D4D3FQV4XUDZQ, seg_pos={0 0}, upload=509109]: Done uploading piece. err: <nil> Dec 29 20:15:01.613856723 (580h46m1.484939768s) [noise=true , address=145.79.227.51:4411 , piece_id=LYCY2Z7Z3BFR5XSM6JXFCL5LZAU56PUBVGHLVURZUTQTEZF7CLFQ, seg_pos={0 0}, upload=509109]: Done uploading piece. err: <nil> Dec 29 20:15:01.615434938 (580h46m1.486517973s) [noise=true , address=145.79.227.46:4602 , piece_id=ZJA75URLP6YW2ROXGLR4PONX2PXXWZNGGANAI37SREOPJ5S7OV3A, seg_pos={0 0}, upload=509109]: Done uploading piece. err: <nil> Dec 29 20:15:01.617581145 (580h46m1.48866419s) [noise=true , address=145.79.227.19:4078 , piece_id=NZLVPFPYJXZHMK2XSIMIDBR7CN4OWVWT5QGNLTK3STC5KBY7CMGA, seg_pos={0 0}, upload=509109]: Done uploading piece. err: <nil> Dec 29 20:15:01.618039818 (580h46m1.489122863s) [noise=true , address=145.79.227.15:4875 , piece_id=RFIPM3MTYFN5PLV4NVNF7AE5OVA54ZIOR2PUY7S4674CN466ECBA, seg_pos={0 0}, upload=509109]: Done uploading piece. err: <nil> Dec 29 20:15:01.624408906 (580h46m1.495491951s) [noise=true , address=145.79.227.36:4518 , piece_id=3DYS7R2VTEUGCDWBWFP2OZHYTMQGLK7OS3OFSCUBIKWSWHJ44ELQ, seg_pos={0 0}, upload=509109]: Done uploading piece. err: <nil> Dec 29 20:15:01.629708417 (580h46m1.500791482s) [noise=true , address=145.79.227.2:4115 , piece_id=J6ZAJU7FYPAFEI47EJBUUUCASWZY2PZZWABSA7WRLAQPYQMFTENQ, seg_pos={0 0}, upload=509109]: Done uploading piece. err: <nil> Dec 29 20:15:01.654622637 (580h46m1.525705682s) [noise=true , address=145.79.227.18:4006 , piece_id=QP3MYNJN7EGPAM7NB4E2ITEV6EPATVAIWGAPT3KNEUXEPPBNQVDA, seg_pos={0 0}, upload=509109]: Done uploading piece. err: <nil> Dec 29 20:15:01.799321647 (580h46m1.670404682s) [noise=true , address=77.90.31.13:4627 , piece_id=OVJXRA53CFJFZSRJ6H2MHZLO4RCLLC3TIGGYQN3ESNQFA3ESQD3A, seg_pos={0 0}, upload=509109]: Done uploading piece. err: <nil> Dec 29 20:15:01.902753888 (580h46m1.773836943s) [noise=true , address=45.91.116.7:4010 , piece_id=BO772OMWJBZDCBAX3O6ECC766FEHZVTOJHKJHDLCSXJWKEKIV6BA, seg_pos={0 0}, upload=509109]: Done uploading piece. err: <nil> Dec 29 20:15:02.167078173 (580h46m2.038161228s) [noise=true , address=45.91.116.9:4262 , piece_id=UV5XMT447DE6QTJYRYBD7466OPDQ3YQSIVQ4C2BSIO37OCZKGSHA, seg_pos={0 0}, upload=509109]: Done uploading piece. err: <nil> Dec 29 20:15:02.167162874 (580h46m2.038245929s) [seg_pos={0 0}, upload=509109]: Segment reached optimal threshold of 13 pieces. Dec 29 20:15:02.170200207 (580h46m2.041283262s) [noise=true , address=45.91.116.6:4975 , piece_id=YHNIUJWPFG4EC7FJYL43CCVFTKZ2GMKX5ANKEIHDMLAL2AGA7V3A, seg_pos={0 0}, upload=509109]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: read tcp 188.166.225.170:42964->45.91.116.6:4975: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 29 20:15:02.170215235 (580h46m2.04129829s) [noise=true , address=45.91.116.6:4975 , piece_id=YHNIUJWPFG4EC7FJYL43CCVFTKZ2GMKX5ANKEIHDMLAL2AGA7V3A, seg_pos={0 0}, upload=509109]: Long tail cancellation due to optimal threshold Dec 29 20:15:02.170672046 (580h46m2.041755101s) [noise=true , address=145.79.227.44:4194 , piece_id=NEQVB6ERVDONQCXWRFXXFPZ3EBSLRJJMWOZHSBCKZWFWZ5FJXTNA, seg_pos={0 0}, upload=509109]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh): protocol: expected piece hash; context canceled; EOF Dec 29 20:15:02.170683507 (580h46m2.041766553s) [noise=true , address=145.79.227.44:4194 , piece_id=NEQVB6ERVDONQCXWRFXXFPZ3EBSLRJJMWOZHSBCKZWFWZ5FJXTNA, seg_pos={0 0}, upload=509109]: Long tail cancellation due to optimal threshold Dec 29 20:15:02.170846298 (580h46m2.041929353s) [noise=true , address=145.79.227.28:4725 , piece_id=U442UUBRULRA3AY6H7EP5G2XJFY47YYDMBWRCMQBM2BLCKZJBYDA, seg_pos={0 0}, upload=509109]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12YKLTah3swX14kL82eNogJxd8D8Hjfq7SPjDDQAapDnXmDctP1): protocol: write tcp 188.166.225.170:56476->145.79.227.28:4725: use of closed network connection; write tcp 188.166.225.170:56476->145.79.227.28:4725: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56476->145.79.227.28:4725: use of closed network connection Dec 29 20:15:02.170859112 (580h46m2.041942157s) [noise=true , address=145.79.227.28:4725 , piece_id=U442UUBRULRA3AY6H7EP5G2XJFY47YYDMBWRCMQBM2BLCKZJBYDA, seg_pos={0 0}, upload=509109]: Long tail cancellation due to optimal threshold Dec 29 20:15:02.170884811 (580h46m2.041967856s) [seg_pos={0 0}, upload=509109]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 20:15:02.171053743 (580h46m2.042136788s) [upload=509109]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 20:15:02.171062129 (580h46m2.042145184s) [upload=509109]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 20:15:02.455142652 (580h46m2.326225707s) [upload=509109]: Done uploading segments... Dec 29 20:15:02.455171597 (580h46m2.326254642s) [upload=509109]: Upload finished. err: <nil> Duration: 6.097605189s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [X..XXX.XX...X.X....X..X.X....X.......X.....................X] duration: 30.002515786s trace: https://jaeger.dev.storj.io/trace/607cf963331dd581 FAIL (2025-12-29 20:15:28 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 20:14:59.343887631 (580h45m59.214970666s) [upload=509109]: Done starting upload Dec 29 20:14:59.343929450 (580h45m59.215012485s) [upload=509109]: Uploading segments... Dec 29 20:14:59.344178725 (580h45m59.21526176s) [upload=509109]: Got next segment. Inline: false Dec 29 20:14:59.344213341 (580h45m59.215296376s) [upload=509109]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 20:14:59.344218040 (580h45m59.215301075s) [upload=509109]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 20:14:59.620876250 (580h45m59.491959285s) [seg_pos={0 0}, upload=509109]: Begin upload segment... Dec 29 20:14:59.620932867 (580h45m59.492015912s) [seg_pos={0 0}, upload=509109]: Done begin upload segment. Dec 29 20:14:59.620975569 (580h45m59.492058604s) [noise=true , address=145.79.227.51:4411 , piece_id=LYCY2Z7Z3BFR5XSM6JXFCL5LZAU56PUBVGHLVURZUTQTEZF7CLFQ, seg_pos={0 0}, upload=509109]: Uploading piece... Dec 29 20:14:59.620983133 (580h45m59.492066178s) [noise=true , address=145.79.227.48:4303 , piece_id=UYEJZYH7QYLL6KGKEQBH6DEGEBMWZSM2OT7W4V6D4D3FQV4XUDZQ, seg_pos={0 0}, upload=509109]: Uploading piece... Dec 29 20:14:59.621128750 (580h45m59.492211796s) [noise=true , address=145.79.227.46:4602 , piece_id=ZJA75URLP6YW2ROXGLR4PONX2PXXWZNGGANAI37SREOPJ5S7OV3A, seg_pos={0 0}, upload=509109]: Uploading piece... Dec 29 20:14:59.621771063 (580h45m59.492854098s) [noise=true , address=145.79.227.15:4875 , piece_id=RFIPM3MTYFN5PLV4NVNF7AE5OVA54ZIOR2PUY7S4674CN466ECBA, seg_pos={0 0}, upload=509109]: Uploading piece... Dec 29 20:14:59.622036208 (580h45m59.493119263s) [noise=true , address=145.79.227.18:4006 , piece_id=QP3MYNJN7EGPAM7NB4E2ITEV6EPATVAIWGAPT3KNEUXEPPBNQVDA, seg_pos={0 0}, upload=509109]: Uploading piece... Dec 29 20:14:59.622303207 (580h45m59.493386252s) [noise=true , address=145.79.227.28:4725 , piece_id=U442UUBRULRA3AY6H7EP5G2XJFY47YYDMBWRCMQBM2BLCKZJBYDA, seg_pos={0 0}, upload=509109]: Uploading piece... Dec 29 20:14:59.622693610 (580h45m59.493776645s) [noise=true , address=145.79.227.44:4194 , piece_id=NEQVB6ERVDONQCXWRFXXFPZ3EBSLRJJMWOZHSBCKZWFWZ5FJXTNA, seg_pos={0 0}, upload=509109]: Uploading piece... Dec 29 20:14:59.623383705 (580h45m59.49446674s) [noise=true , address=77.90.31.13:4627 , piece_id=OVJXRA53CFJFZSRJ6H2MHZLO4RCLLC3TIGGYQN3ESNQFA3ESQD3A, seg_pos={0 0}, upload=509109]: Uploading piece... Dec 29 20:14:59.623912161 (580h45m59.494995206s) [noise=true , address=45.91.116.7:4010 , piece_id=BO772OMWJBZDCBAX3O6ECC766FEHZVTOJHKJHDLCSXJWKEKIV6BA, seg_pos={0 0}, upload=509109]: Uploading piece... Dec 29 20:14:59.624389991 (580h45m59.495473026s) [noise=true , address=145.79.227.2:4115 , piece_id=J6ZAJU7FYPAFEI47EJBUUUCASWZY2PZZWABSA7WRLAQPYQMFTENQ, seg_pos={0 0}, upload=509109]: Uploading piece... Dec 29 20:14:59.624497966 (580h45m59.495581001s) [noise=true , address=145.79.227.11:4731 , piece_id=46XM74D3UEXNZXHZYWFOYW6SHHWTQU2HA6Q2WYSJLH7OVC23JALQ, seg_pos={0 0}, upload=509109]: Uploading piece... Dec 29 20:14:59.624935829 (580h45m59.496018864s) [noise=true , address=145.79.227.19:4078 , piece_id=NZLVPFPYJXZHMK2XSIMIDBR7CN4OWVWT5QGNLTK3STC5KBY7CMGA, seg_pos={0 0}, upload=509109]: Uploading piece... Dec 29 20:14:59.624977129 (580h45m59.496060174s) [noise=true , address=45.91.116.9:4262 , piece_id=UV5XMT447DE6QTJYRYBD7466OPDQ3YQSIVQ4C2BSIO37OCZKGSHA, seg_pos={0 0}, upload=509109]: Uploading piece... Dec 29 20:14:59.625338717 (580h45m59.496421762s) [noise=true , address=45.91.116.6:4975 , piece_id=YHNIUJWPFG4EC7FJYL43CCVFTKZ2GMKX5ANKEIHDMLAL2AGA7V3A, seg_pos={0 0}, upload=509109]: Uploading piece... Dec 29 20:14:59.625486658 (580h45m59.496569704s) [noise=true , address=145.79.227.36:4518 , piece_id=3DYS7R2VTEUGCDWBWFP2OZHYTMQGLK7OS3OFSCUBIKWSWHJ44ELQ, seg_pos={0 0}, upload=509109]: Uploading piece... Dec 29 20:14:59.625852676 (580h45m59.496935721s) [noise=true , address=145.79.227.9:4366 , piece_id=ICYZM4QQZQJLS4T4KYM47NEMIKYD7PZWQ2U2NEFLAWLLPO6OFG3A, seg_pos={0 0}, upload=509109]: Uploading piece... Dec 29 20:15:01.223830613 (580h46m1.094913658s) [upload=509109]: Next returned nil segment Dec 29 20:15:01.223865600 (580h46m1.094948645s) [upload=509109]: Waiting for error group managing segments... Dec 29 20:15:01.599999537 (580h46m1.471082582s) [noise=true , address=145.79.227.11:4731 , piece_id=46XM74D3UEXNZXHZYWFOYW6SHHWTQU2HA6Q2WYSJLH7OVC23JALQ, seg_pos={0 0}, upload=509109]: Done uploading piece. err: <nil> Dec 29 20:15:01.606734191 (580h46m1.477817236s) [noise=true , address=145.79.227.9:4366 , piece_id=ICYZM4QQZQJLS4T4KYM47NEMIKYD7PZWQ2U2NEFLAWLLPO6OFG3A, seg_pos={0 0}, upload=509109]: Done uploading piece. err: <nil> Dec 29 20:15:01.609790219 (580h46m1.480873265s) [noise=true , address=145.79.227.48:4303 , piece_id=UYEJZYH7QYLL6KGKEQBH6DEGEBMWZSM2OT7W4V6D4D3FQV4XUDZQ, seg_pos={0 0}, upload=509109]: Done uploading piece. err: <nil> Dec 29 20:15:01.613856723 (580h46m1.484939768s) [noise=true , address=145.79.227.51:4411 , piece_id=LYCY2Z7Z3BFR5XSM6JXFCL5LZAU56PUBVGHLVURZUTQTEZF7CLFQ, seg_pos={0 0}, upload=509109]: Done uploading piece. err: <nil> Dec 29 20:15:01.615434938 (580h46m1.486517973s) [noise=true , address=145.79.227.46:4602 , piece_id=ZJA75URLP6YW2ROXGLR4PONX2PXXWZNGGANAI37SREOPJ5S7OV3A, seg_pos={0 0}, upload=509109]: Done uploading piece. err: <nil> Dec 29 20:15:01.617581145 (580h46m1.48866419s) [noise=true , address=145.79.227.19:4078 , piece_id=NZLVPFPYJXZHMK2XSIMIDBR7CN4OWVWT5QGNLTK3STC5KBY7CMGA, seg_pos={0 0}, upload=509109]: Done uploading piece. err: <nil> Dec 29 20:15:01.618039818 (580h46m1.489122863s) [noise=true , address=145.79.227.15:4875 , piece_id=RFIPM3MTYFN5PLV4NVNF7AE5OVA54ZIOR2PUY7S4674CN466ECBA, seg_pos={0 0}, upload=509109]: Done uploading piece. err: <nil> Dec 29 20:15:01.624408906 (580h46m1.495491951s) [noise=true , address=145.79.227.36:4518 , piece_id=3DYS7R2VTEUGCDWBWFP2OZHYTMQGLK7OS3OFSCUBIKWSWHJ44ELQ, seg_pos={0 0}, upload=509109]: Done uploading piece. err: <nil> Dec 29 20:15:01.629708417 (580h46m1.500791482s) [noise=true , address=145.79.227.2:4115 , piece_id=J6ZAJU7FYPAFEI47EJBUUUCASWZY2PZZWABSA7WRLAQPYQMFTENQ, seg_pos={0 0}, upload=509109]: Done uploading piece. err: <nil> Dec 29 20:15:01.654622637 (580h46m1.525705682s) [noise=true , address=145.79.227.18:4006 , piece_id=QP3MYNJN7EGPAM7NB4E2ITEV6EPATVAIWGAPT3KNEUXEPPBNQVDA, seg_pos={0 0}, upload=509109]: Done uploading piece. err: <nil> Dec 29 20:15:01.799321647 (580h46m1.670404682s) [noise=true , address=77.90.31.13:4627 , piece_id=OVJXRA53CFJFZSRJ6H2MHZLO4RCLLC3TIGGYQN3ESNQFA3ESQD3A, seg_pos={0 0}, upload=509109]: Done uploading piece. err: <nil> Dec 29 20:15:01.902753888 (580h46m1.773836943s) [noise=true , address=45.91.116.7:4010 , piece_id=BO772OMWJBZDCBAX3O6ECC766FEHZVTOJHKJHDLCSXJWKEKIV6BA, seg_pos={0 0}, upload=509109]: Done uploading piece. err: <nil> Dec 29 20:15:02.167078173 (580h46m2.038161228s) [noise=true , address=45.91.116.9:4262 , piece_id=UV5XMT447DE6QTJYRYBD7466OPDQ3YQSIVQ4C2BSIO37OCZKGSHA, seg_pos={0 0}, upload=509109]: Done uploading piece. err: <nil> Dec 29 20:15:02.167162874 (580h46m2.038245929s) [seg_pos={0 0}, upload=509109]: Segment reached optimal threshold of 13 pieces. Dec 29 20:15:02.170200207 (580h46m2.041283262s) [noise=true , address=45.91.116.6:4975 , piece_id=YHNIUJWPFG4EC7FJYL43CCVFTKZ2GMKX5ANKEIHDMLAL2AGA7V3A, seg_pos={0 0}, upload=509109]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: read tcp 188.166.225.170:42964->45.91.116.6:4975: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 29 20:15:02.170215235 (580h46m2.04129829s) [noise=true , address=45.91.116.6:4975 , piece_id=YHNIUJWPFG4EC7FJYL43CCVFTKZ2GMKX5ANKEIHDMLAL2AGA7V3A, seg_pos={0 0}, upload=509109]: Long tail cancellation due to optimal threshold Dec 29 20:15:02.170672046 (580h46m2.041755101s) [noise=true , address=145.79.227.44:4194 , piece_id=NEQVB6ERVDONQCXWRFXXFPZ3EBSLRJJMWOZHSBCKZWFWZ5FJXTNA, seg_pos={0 0}, upload=509109]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh): protocol: expected piece hash; context canceled; EOF Dec 29 20:15:02.170683507 (580h46m2.041766553s) [noise=true , address=145.79.227.44:4194 , piece_id=NEQVB6ERVDONQCXWRFXXFPZ3EBSLRJJMWOZHSBCKZWFWZ5FJXTNA, seg_pos={0 0}, upload=509109]: Long tail cancellation due to optimal threshold Dec 29 20:15:02.170846298 (580h46m2.041929353s) [noise=true , address=145.79.227.28:4725 , piece_id=U442UUBRULRA3AY6H7EP5G2XJFY47YYDMBWRCMQBM2BLCKZJBYDA, seg_pos={0 0}, upload=509109]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12YKLTah3swX14kL82eNogJxd8D8Hjfq7SPjDDQAapDnXmDctP1): protocol: write tcp 188.166.225.170:56476->145.79.227.28:4725: use of closed network connection; write tcp 188.166.225.170:56476->145.79.227.28:4725: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56476->145.79.227.28:4725: use of closed network connection Dec 29 20:15:02.170859112 (580h46m2.041942157s) [noise=true , address=145.79.227.28:4725 , piece_id=U442UUBRULRA3AY6H7EP5G2XJFY47YYDMBWRCMQBM2BLCKZJBYDA, seg_pos={0 0}, upload=509109]: Long tail cancellation due to optimal threshold Dec 29 20:15:02.170884811 (580h46m2.041967856s) [seg_pos={0 0}, upload=509109]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 20:15:02.171053743 (580h46m2.042136788s) [upload=509109]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 20:15:02.171062129 (580h46m2.042145184s) [upload=509109]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 20:15:02.455142652 (580h46m2.326225707s) [upload=509109]: Done uploading segments... Dec 29 20:15:02.455171597 (580h46m2.326254642s) [upload=509109]: Upload finished. err: <nil> Duration: 30.002526306s core uplink us-select-1
checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing) antiblip: 1 out of 60 is failed (threshold=6) [.................................X..........................] duration: 248.475205ms remote: "tcp:207.211.208.129:443" target: https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt trace: https://jaeger.dev.storj.io/trace/4679501fce62ca7 FAIL (2025-12-29 20:15:26 UTC) orbiter warning: Get "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt": context deadline exceeded Duration: 248.505021ms edge linksharing bgp eu1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing)

checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing)

checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing)

checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing)

checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing)