Orbiter History

checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 26 out of 60 is failed (threshold=6) [......X.X..X..XX.XXXXXXX.X...X..X.XX.....X.X.XX.X.X.X..X...X] duration: 5.000856322s trace: https://jaeger.dev.storj.io/trace/7bace3abcfd3c380 FAIL (2025-12-28 07:36:35 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000879876s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [XXX.XX.............XX.X...X...X.XXX..X.XX.X.........XX..X...] duration: 5.861602201s trace: https://jaeger.dev.storj.io/trace/133f3e3251246010 FAIL (2025-12-28 07:35:47 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, Piece ID: SHLAKBO2LXCNL34YAV7WE5DBG3OSUEOMNRFGY3DLXLIEBGGZHQCA) context deadline exceeded; piecestore: (Node ID: 12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs, Piece ID: LTQ2GATZ37YL54OTBGYFOYHSTV7GYNZBGVFJ7LLHSHGDVHSTO5IQ) context deadline exceeded ======LOGS====== Dec 28 07:34:37.532705347 (544h5m37.403788382s) [upload=477142]: Starting upload Dec 28 07:34:37.532783416 (544h5m37.403866461s) [upload=477142]: Done starting upload Dec 28 07:34:37.532861695 (544h5m37.40394474s) [upload=477142]: Uploading segments... Dec 28 07:34:37.532979148 (544h5m37.404062183s) [upload=477142]: Got next segment. Inline: false Dec 28 07:34:37.533015638 (544h5m37.404098673s) [upload=477142]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 07:34:37.533020767 (544h5m37.404103812s) [upload=477142]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 07:34:37.778628096 (544h5m37.649711141s) [seg_pos={0 0}, upload=477142]: Begin upload segment... Dec 28 07:34:37.778665327 (544h5m37.649748362s) [seg_pos={0 0}, upload=477142]: Done begin upload segment. Dec 28 07:34:37.778694293 (544h5m37.649777328s) [noise=true , address=145.79.227.14:4839 , piece_id=2SMJMCT2Q4TRJSPBPF37KU5KR6N4W7GTNPM3DOHJH5XFUJGOC3GQ, seg_pos={0 0}, upload=477142]: Uploading piece... Dec 28 07:34:37.779877405 (544h5m37.650960451s) [noise=true , address=145.79.227.39:4014 , piece_id=DTFEKJMIHZYP72HMNCWQUIMJV532PVGB646N3LL7N65OLCLPHXEQ, seg_pos={0 0}, upload=477142]: Uploading piece... Dec 28 07:34:37.780457250 (544h5m37.651540285s) [noise=true , address=145.79.227.44:4194 , piece_id=ORQFOZR4S3QGKNFS3NTLNCY4CN4MB4DUHCQINRGHLMQF72H5D6AA, seg_pos={0 0}, upload=477142]: Uploading piece... Dec 28 07:34:37.780917376 (544h5m37.652000421s) [noise=true , address=145.79.227.48:4301 , piece_id=HHK2NW5UQFHQ2SOGPCE4KNSMYYIFLQKZ5HMGJHMOZDHT6U7DJTHQ, seg_pos={0 0}, upload=477142]: Uploading piece... Dec 28 07:34:37.781384245 (544h5m37.652467291s) [noise=true , address=145.79.227.38:4977 , piece_id=NKZREBCYULHBBQJUEZJOBEO4L2FHMSYBB4L4D5777JE6DVDWOFPQ, seg_pos={0 0}, upload=477142]: Uploading piece... Dec 28 07:34:37.781762274 (544h5m37.652845319s) [noise=true , address=145.79.227.7:4295 , piece_id=2LMCBOGQZNLJFUUDOEKDNSB2JBUTWGVXU5E7XEUSAJN2RKZFNULQ, seg_pos={0 0}, upload=477142]: Uploading piece... Dec 28 07:34:37.782192785 (544h5m37.65327582s) [noise=true , address=45.91.116.4:4939 , piece_id=K4PM7J7FNH7RFN3BN42MYNNS5YXKVTL7QHXIBPG6CLFFTUEGF7EA, seg_pos={0 0}, upload=477142]: Uploading piece... Dec 28 07:34:37.782622242 (544h5m37.653705277s) [noise=true , address=145.79.227.42:4121 , piece_id=JY52UW3TWMFP2X3YWQOWIHWWVLX2TRQXVS3C2TEBLOTDQGK2FMGA, seg_pos={0 0}, upload=477142]: Uploading piece... Dec 28 07:34:37.782986907 (544h5m37.654069942s) [noise=true , address=145.79.227.23:4547 , piece_id=IBUYL6SJ6BK6HPZ6MWXOCDUZX6RGFSQCCWRGLDTFLH4PNFNOVVZA, seg_pos={0 0}, upload=477142]: Uploading piece... Dec 28 07:34:37.783358294 (544h5m37.654441329s) [noise=true , address=145.79.227.16:4909 , piece_id=AHT57DVJN62EZ35M7VOR3Q6WXFILPYRQIX44VICANKC7FYR5IMTQ, seg_pos={0 0}, upload=477142]: Uploading piece... Dec 28 07:34:37.783741303 (544h5m37.654824348s) [noise=true , address=145.79.227.41:4086 , piece_id=D5JRM2DFKV75RLZQNKGIT4W5WXU5L4DDD6YDW3GZF5EUFMNIVF4A, seg_pos={0 0}, upload=477142]: Uploading piece... Dec 28 07:34:37.784137518 (544h5m37.655220563s) [noise=true , address=145.79.227.19:4078 , piece_id=2CFG67OTHVOLV2FLRBN54T5A77RM7XEGMG3TDZUXGBXIICSHJMZA, seg_pos={0 0}, upload=477142]: Uploading piece... Dec 28 07:34:37.784483848 (544h5m37.655566883s) [noise=true , address=45.91.116.8:4045 , piece_id=SHLAKBO2LXCNL34YAV7WE5DBG3OSUEOMNRFGY3DLXLIEBGGZHQCA, seg_pos={0 0}, upload=477142]: Uploading piece... Dec 28 07:34:37.784772717 (544h5m37.655855752s) [noise=true , address=145.79.227.2:4113 , piece_id=CYODBMBDDCCTQJK4CSPCZI43WLMK5D6HCXPUCVPXH4JHRES7XVVA, seg_pos={0 0}, upload=477142]: Uploading piece... Dec 28 07:34:37.785084731 (544h5m37.656167767s) [noise=true , address=45.91.116.9:4263 , piece_id=HFLF2R732JKFUEZPELMGNXFCI7SCNGC3HEHP4ZAK6EIH2BHZRUDQ, seg_pos={0 0}, upload=477142]: Uploading piece... Dec 28 07:34:37.785502818 (544h5m37.656585853s) [noise=true , address=45.91.116.14:4192 , piece_id=LTQ2GATZ37YL54OTBGYFOYHSTV7GYNZBGVFJ7LLHSHGDVHSTO5IQ, seg_pos={0 0}, upload=477142]: Uploading piece... Dec 28 07:34:39.496817391 (544h5m39.367900426s) [upload=477142]: Next returned nil segment Dec 28 07:34:39.496834414 (544h5m39.367917459s) [upload=477142]: Waiting for error group managing segments... Dec 28 07:34:39.782997869 (544h5m39.654080904s) [noise=true , address=145.79.227.38:4977 , piece_id=NKZREBCYULHBBQJUEZJOBEO4L2FHMSYBB4L4D5777JE6DVDWOFPQ, seg_pos={0 0}, upload=477142]: Done uploading piece. err: <nil> Dec 28 07:34:39.788982756 (544h5m39.660065801s) [noise=true , address=145.79.227.14:4839 , piece_id=2SMJMCT2Q4TRJSPBPF37KU5KR6N4W7GTNPM3DOHJH5XFUJGOC3GQ, seg_pos={0 0}, upload=477142]: Done uploading piece. err: <nil> Dec 28 07:34:39.793466995 (544h5m39.66455004s) [noise=true , address=145.79.227.19:4078 , piece_id=2CFG67OTHVOLV2FLRBN54T5A77RM7XEGMG3TDZUXGBXIICSHJMZA, seg_pos={0 0}, upload=477142]: Done uploading piece. err: <nil> Dec 28 07:34:39.794004298 (544h5m39.665087333s) [noise=true , address=145.79.227.41:4086 , piece_id=D5JRM2DFKV75RLZQNKGIT4W5WXU5L4DDD6YDW3GZF5EUFMNIVF4A, seg_pos={0 0}, upload=477142]: Done uploading piece. err: <nil> Dec 28 07:34:39.795054618 (544h5m39.666137663s) [noise=true , address=145.79.227.39:4014 , piece_id=DTFEKJMIHZYP72HMNCWQUIMJV532PVGB646N3LL7N65OLCLPHXEQ, seg_pos={0 0}, upload=477142]: Done uploading piece. err: <nil> Dec 28 07:34:39.799205542 (544h5m39.670288577s) [noise=true , address=145.79.227.7:4295 , piece_id=2LMCBOGQZNLJFUUDOEKDNSB2JBUTWGVXU5E7XEUSAJN2RKZFNULQ, seg_pos={0 0}, upload=477142]: Done uploading piece. err: <nil> Dec 28 07:34:39.799745961 (544h5m39.670828996s) [noise=true , address=145.79.227.2:4113 , piece_id=CYODBMBDDCCTQJK4CSPCZI43WLMK5D6HCXPUCVPXH4JHRES7XVVA, seg_pos={0 0}, upload=477142]: Done uploading piece. err: <nil> Dec 28 07:34:39.810738302 (544h5m39.681821337s) [noise=true , address=145.79.227.48:4301 , piece_id=HHK2NW5UQFHQ2SOGPCE4KNSMYYIFLQKZ5HMGJHMOZDHT6U7DJTHQ, seg_pos={0 0}, upload=477142]: Done uploading piece. err: <nil> Dec 28 07:34:39.814975691 (544h5m39.686058736s) [noise=true , address=145.79.227.23:4547 , piece_id=IBUYL6SJ6BK6HPZ6MWXOCDUZX6RGFSQCCWRGLDTFLH4PNFNOVVZA, seg_pos={0 0}, upload=477142]: Done uploading piece. err: <nil> Dec 28 07:34:39.816049686 (544h5m39.687132731s) [noise=true , address=145.79.227.16:4909 , piece_id=AHT57DVJN62EZ35M7VOR3Q6WXFILPYRQIX44VICANKC7FYR5IMTQ, seg_pos={0 0}, upload=477142]: Done uploading piece. err: <nil> Dec 28 07:34:39.823296435 (544h5m39.69437948s) [noise=true , address=145.79.227.42:4121 , piece_id=JY52UW3TWMFP2X3YWQOWIHWWVLX2TRQXVS3C2TEBLOTDQGK2FMGA, seg_pos={0 0}, upload=477142]: Done uploading piece. err: <nil> Dec 28 07:34:40.133349220 (544h5m40.004432265s) [noise=true , address=45.91.116.14:4192 , piece_id=LTQ2GATZ37YL54OTBGYFOYHSTV7GYNZBGVFJ7LLHSHGDVHSTO5IQ, seg_pos={0 0}, upload=477142]: Done uploading piece. err: <nil> Dec 28 07:34:40.159334831 (544h5m40.030417866s) [noise=true , address=45.91.116.8:4045 , piece_id=SHLAKBO2LXCNL34YAV7WE5DBG3OSUEOMNRFGY3DLXLIEBGGZHQCA, seg_pos={0 0}, upload=477142]: Done uploading piece. err: <nil> Dec 28 07:34:40.159355731 (544h5m40.030438776s) [seg_pos={0 0}, upload=477142]: Segment reached optimal threshold of 13 pieces. Dec 28 07:34:40.159548138 (544h5m40.030631183s) [noise=true , address=45.91.116.4:4939 , piece_id=K4PM7J7FNH7RFN3BN42MYNNS5YXKVTL7QHXIBPG6CLFFTUEGF7EA, seg_pos={0 0}, upload=477142]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: expected piece hash; context canceled; EOF Dec 28 07:34:40.159554710 (544h5m40.030637765s) [noise=true , address=45.91.116.4:4939 , piece_id=K4PM7J7FNH7RFN3BN42MYNNS5YXKVTL7QHXIBPG6CLFFTUEGF7EA, seg_pos={0 0}, upload=477142]: Long tail cancellation due to optimal threshold Dec 28 07:34:40.159709835 (544h5m40.03079287s) [noise=true , address=145.79.227.44:4194 , piece_id=ORQFOZR4S3QGKNFS3NTLNCY4CN4MB4DUHCQINRGHLMQF72H5D6AA, seg_pos={0 0}, upload=477142]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh): protocol: expected piece hash; context canceled; EOF Dec 28 07:34:40.159717720 (544h5m40.030800755s) [noise=true , address=145.79.227.44:4194 , piece_id=ORQFOZR4S3QGKNFS3NTLNCY4CN4MB4DUHCQINRGHLMQF72H5D6AA, seg_pos={0 0}, upload=477142]: Long tail cancellation due to optimal threshold Dec 28 07:34:40.159785700 (544h5m40.030868735s) [noise=true , address=45.91.116.9:4263 , piece_id=HFLF2R732JKFUEZPELMGNXFCI7SCNGC3HEHP4ZAK6EIH2BHZRUDQ, seg_pos={0 0}, upload=477142]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF): protocol: expected piece hash; context canceled; EOF Dec 28 07:34:40.159795018 (544h5m40.030878063s) [noise=true , address=45.91.116.9:4263 , piece_id=HFLF2R732JKFUEZPELMGNXFCI7SCNGC3HEHP4ZAK6EIH2BHZRUDQ, seg_pos={0 0}, upload=477142]: Long tail cancellation due to optimal threshold Dec 28 07:34:40.159819514 (544h5m40.030902559s) [seg_pos={0 0}, upload=477142]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 07:34:40.159906220 (544h5m40.030989265s) [upload=477142]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 07:34:40.159915136 (544h5m40.030998201s) [upload=477142]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 07:34:40.457024204 (544h5m40.328107249s) [upload=477142]: Done uploading segments... Dec 28 07:34:40.457045084 (544h5m40.32812812s) [upload=477142]: Upload finished. err: <nil> Duration: 5.861626576s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [.XXX.XX.............XX.X...X...X.XXX..X.XX.X.........XX..X..] duration: 11.200666847s trace: https://jaeger.dev.storj.io/trace/133f3e3251246010 FAIL (2025-12-28 07:35:36 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, Piece ID: SHLAKBO2LXCNL34YAV7WE5DBG3OSUEOMNRFGY3DLXLIEBGGZHQCA) context deadline exceeded; piecestore: (Node ID: 12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs, Piece ID: LTQ2GATZ37YL54OTBGYFOYHSTV7GYNZBGVFJ7LLHSHGDVHSTO5IQ) context deadline exceeded ======LOGS====== Dec 28 07:34:37.532705347 (544h5m37.403788382s) [upload=477142]: Starting upload Dec 28 07:34:37.532783416 (544h5m37.403866461s) [upload=477142]: Done starting upload Dec 28 07:34:37.532861695 (544h5m37.40394474s) [upload=477142]: Uploading segments... Dec 28 07:34:37.532979148 (544h5m37.404062183s) [upload=477142]: Got next segment. Inline: false Dec 28 07:34:37.533015638 (544h5m37.404098673s) [upload=477142]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 07:34:37.533020767 (544h5m37.404103812s) [upload=477142]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 07:34:37.778628096 (544h5m37.649711141s) [seg_pos={0 0}, upload=477142]: Begin upload segment... Dec 28 07:34:37.778665327 (544h5m37.649748362s) [seg_pos={0 0}, upload=477142]: Done begin upload segment. Dec 28 07:34:37.778694293 (544h5m37.649777328s) [noise=true , address=145.79.227.14:4839 , piece_id=2SMJMCT2Q4TRJSPBPF37KU5KR6N4W7GTNPM3DOHJH5XFUJGOC3GQ, seg_pos={0 0}, upload=477142]: Uploading piece... Dec 28 07:34:37.779877405 (544h5m37.650960451s) [noise=true , address=145.79.227.39:4014 , piece_id=DTFEKJMIHZYP72HMNCWQUIMJV532PVGB646N3LL7N65OLCLPHXEQ, seg_pos={0 0}, upload=477142]: Uploading piece... Dec 28 07:34:37.780457250 (544h5m37.651540285s) [noise=true , address=145.79.227.44:4194 , piece_id=ORQFOZR4S3QGKNFS3NTLNCY4CN4MB4DUHCQINRGHLMQF72H5D6AA, seg_pos={0 0}, upload=477142]: Uploading piece... Dec 28 07:34:37.780917376 (544h5m37.652000421s) [noise=true , address=145.79.227.48:4301 , piece_id=HHK2NW5UQFHQ2SOGPCE4KNSMYYIFLQKZ5HMGJHMOZDHT6U7DJTHQ, seg_pos={0 0}, upload=477142]: Uploading piece... Dec 28 07:34:37.781384245 (544h5m37.652467291s) [noise=true , address=145.79.227.38:4977 , piece_id=NKZREBCYULHBBQJUEZJOBEO4L2FHMSYBB4L4D5777JE6DVDWOFPQ, seg_pos={0 0}, upload=477142]: Uploading piece... Dec 28 07:34:37.781762274 (544h5m37.652845319s) [noise=true , address=145.79.227.7:4295 , piece_id=2LMCBOGQZNLJFUUDOEKDNSB2JBUTWGVXU5E7XEUSAJN2RKZFNULQ, seg_pos={0 0}, upload=477142]: Uploading piece... Dec 28 07:34:37.782192785 (544h5m37.65327582s) [noise=true , address=45.91.116.4:4939 , piece_id=K4PM7J7FNH7RFN3BN42MYNNS5YXKVTL7QHXIBPG6CLFFTUEGF7EA, seg_pos={0 0}, upload=477142]: Uploading piece... Dec 28 07:34:37.782622242 (544h5m37.653705277s) [noise=true , address=145.79.227.42:4121 , piece_id=JY52UW3TWMFP2X3YWQOWIHWWVLX2TRQXVS3C2TEBLOTDQGK2FMGA, seg_pos={0 0}, upload=477142]: Uploading piece... Dec 28 07:34:37.782986907 (544h5m37.654069942s) [noise=true , address=145.79.227.23:4547 , piece_id=IBUYL6SJ6BK6HPZ6MWXOCDUZX6RGFSQCCWRGLDTFLH4PNFNOVVZA, seg_pos={0 0}, upload=477142]: Uploading piece... Dec 28 07:34:37.783358294 (544h5m37.654441329s) [noise=true , address=145.79.227.16:4909 , piece_id=AHT57DVJN62EZ35M7VOR3Q6WXFILPYRQIX44VICANKC7FYR5IMTQ, seg_pos={0 0}, upload=477142]: Uploading piece... Dec 28 07:34:37.783741303 (544h5m37.654824348s) [noise=true , address=145.79.227.41:4086 , piece_id=D5JRM2DFKV75RLZQNKGIT4W5WXU5L4DDD6YDW3GZF5EUFMNIVF4A, seg_pos={0 0}, upload=477142]: Uploading piece... Dec 28 07:34:37.784137518 (544h5m37.655220563s) [noise=true , address=145.79.227.19:4078 , piece_id=2CFG67OTHVOLV2FLRBN54T5A77RM7XEGMG3TDZUXGBXIICSHJMZA, seg_pos={0 0}, upload=477142]: Uploading piece... Dec 28 07:34:37.784483848 (544h5m37.655566883s) [noise=true , address=45.91.116.8:4045 , piece_id=SHLAKBO2LXCNL34YAV7WE5DBG3OSUEOMNRFGY3DLXLIEBGGZHQCA, seg_pos={0 0}, upload=477142]: Uploading piece... Dec 28 07:34:37.784772717 (544h5m37.655855752s) [noise=true , address=145.79.227.2:4113 , piece_id=CYODBMBDDCCTQJK4CSPCZI43WLMK5D6HCXPUCVPXH4JHRES7XVVA, seg_pos={0 0}, upload=477142]: Uploading piece... Dec 28 07:34:37.785084731 (544h5m37.656167767s) [noise=true , address=45.91.116.9:4263 , piece_id=HFLF2R732JKFUEZPELMGNXFCI7SCNGC3HEHP4ZAK6EIH2BHZRUDQ, seg_pos={0 0}, upload=477142]: Uploading piece... Dec 28 07:34:37.785502818 (544h5m37.656585853s) [noise=true , address=45.91.116.14:4192 , piece_id=LTQ2GATZ37YL54OTBGYFOYHSTV7GYNZBGVFJ7LLHSHGDVHSTO5IQ, seg_pos={0 0}, upload=477142]: Uploading piece... Dec 28 07:34:39.496817391 (544h5m39.367900426s) [upload=477142]: Next returned nil segment Dec 28 07:34:39.496834414 (544h5m39.367917459s) [upload=477142]: Waiting for error group managing segments... Dec 28 07:34:39.782997869 (544h5m39.654080904s) [noise=true , address=145.79.227.38:4977 , piece_id=NKZREBCYULHBBQJUEZJOBEO4L2FHMSYBB4L4D5777JE6DVDWOFPQ, seg_pos={0 0}, upload=477142]: Done uploading piece. err: <nil> Dec 28 07:34:39.788982756 (544h5m39.660065801s) [noise=true , address=145.79.227.14:4839 , piece_id=2SMJMCT2Q4TRJSPBPF37KU5KR6N4W7GTNPM3DOHJH5XFUJGOC3GQ, seg_pos={0 0}, upload=477142]: Done uploading piece. err: <nil> Dec 28 07:34:39.793466995 (544h5m39.66455004s) [noise=true , address=145.79.227.19:4078 , piece_id=2CFG67OTHVOLV2FLRBN54T5A77RM7XEGMG3TDZUXGBXIICSHJMZA, seg_pos={0 0}, upload=477142]: Done uploading piece. err: <nil> Dec 28 07:34:39.794004298 (544h5m39.665087333s) [noise=true , address=145.79.227.41:4086 , piece_id=D5JRM2DFKV75RLZQNKGIT4W5WXU5L4DDD6YDW3GZF5EUFMNIVF4A, seg_pos={0 0}, upload=477142]: Done uploading piece. err: <nil> Dec 28 07:34:39.795054618 (544h5m39.666137663s) [noise=true , address=145.79.227.39:4014 , piece_id=DTFEKJMIHZYP72HMNCWQUIMJV532PVGB646N3LL7N65OLCLPHXEQ, seg_pos={0 0}, upload=477142]: Done uploading piece. err: <nil> Dec 28 07:34:39.799205542 (544h5m39.670288577s) [noise=true , address=145.79.227.7:4295 , piece_id=2LMCBOGQZNLJFUUDOEKDNSB2JBUTWGVXU5E7XEUSAJN2RKZFNULQ, seg_pos={0 0}, upload=477142]: Done uploading piece. err: <nil> Dec 28 07:34:39.799745961 (544h5m39.670828996s) [noise=true , address=145.79.227.2:4113 , piece_id=CYODBMBDDCCTQJK4CSPCZI43WLMK5D6HCXPUCVPXH4JHRES7XVVA, seg_pos={0 0}, upload=477142]: Done uploading piece. err: <nil> Dec 28 07:34:39.810738302 (544h5m39.681821337s) [noise=true , address=145.79.227.48:4301 , piece_id=HHK2NW5UQFHQ2SOGPCE4KNSMYYIFLQKZ5HMGJHMOZDHT6U7DJTHQ, seg_pos={0 0}, upload=477142]: Done uploading piece. err: <nil> Dec 28 07:34:39.814975691 (544h5m39.686058736s) [noise=true , address=145.79.227.23:4547 , piece_id=IBUYL6SJ6BK6HPZ6MWXOCDUZX6RGFSQCCWRGLDTFLH4PNFNOVVZA, seg_pos={0 0}, upload=477142]: Done uploading piece. err: <nil> Dec 28 07:34:39.816049686 (544h5m39.687132731s) [noise=true , address=145.79.227.16:4909 , piece_id=AHT57DVJN62EZ35M7VOR3Q6WXFILPYRQIX44VICANKC7FYR5IMTQ, seg_pos={0 0}, upload=477142]: Done uploading piece. err: <nil> Dec 28 07:34:39.823296435 (544h5m39.69437948s) [noise=true , address=145.79.227.42:4121 , piece_id=JY52UW3TWMFP2X3YWQOWIHWWVLX2TRQXVS3C2TEBLOTDQGK2FMGA, seg_pos={0 0}, upload=477142]: Done uploading piece. err: <nil> Dec 28 07:34:40.133349220 (544h5m40.004432265s) [noise=true , address=45.91.116.14:4192 , piece_id=LTQ2GATZ37YL54OTBGYFOYHSTV7GYNZBGVFJ7LLHSHGDVHSTO5IQ, seg_pos={0 0}, upload=477142]: Done uploading piece. err: <nil> Dec 28 07:34:40.159334831 (544h5m40.030417866s) [noise=true , address=45.91.116.8:4045 , piece_id=SHLAKBO2LXCNL34YAV7WE5DBG3OSUEOMNRFGY3DLXLIEBGGZHQCA, seg_pos={0 0}, upload=477142]: Done uploading piece. err: <nil> Dec 28 07:34:40.159355731 (544h5m40.030438776s) [seg_pos={0 0}, upload=477142]: Segment reached optimal threshold of 13 pieces. Dec 28 07:34:40.159548138 (544h5m40.030631183s) [noise=true , address=45.91.116.4:4939 , piece_id=K4PM7J7FNH7RFN3BN42MYNNS5YXKVTL7QHXIBPG6CLFFTUEGF7EA, seg_pos={0 0}, upload=477142]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: expected piece hash; context canceled; EOF Dec 28 07:34:40.159554710 (544h5m40.030637765s) [noise=true , address=45.91.116.4:4939 , piece_id=K4PM7J7FNH7RFN3BN42MYNNS5YXKVTL7QHXIBPG6CLFFTUEGF7EA, seg_pos={0 0}, upload=477142]: Long tail cancellation due to optimal threshold Dec 28 07:34:40.159709835 (544h5m40.03079287s) [noise=true , address=145.79.227.44:4194 , piece_id=ORQFOZR4S3QGKNFS3NTLNCY4CN4MB4DUHCQINRGHLMQF72H5D6AA, seg_pos={0 0}, upload=477142]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh): protocol: expected piece hash; context canceled; EOF Dec 28 07:34:40.159717720 (544h5m40.030800755s) [noise=true , address=145.79.227.44:4194 , piece_id=ORQFOZR4S3QGKNFS3NTLNCY4CN4MB4DUHCQINRGHLMQF72H5D6AA, seg_pos={0 0}, upload=477142]: Long tail cancellation due to optimal threshold Dec 28 07:34:40.159785700 (544h5m40.030868735s) [noise=true , address=45.91.116.9:4263 , piece_id=HFLF2R732JKFUEZPELMGNXFCI7SCNGC3HEHP4ZAK6EIH2BHZRUDQ, seg_pos={0 0}, upload=477142]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF): protocol: expected piece hash; context canceled; EOF Dec 28 07:34:40.159795018 (544h5m40.030878063s) [noise=true , address=45.91.116.9:4263 , piece_id=HFLF2R732JKFUEZPELMGNXFCI7SCNGC3HEHP4ZAK6EIH2BHZRUDQ, seg_pos={0 0}, upload=477142]: Long tail cancellation due to optimal threshold Dec 28 07:34:40.159819514 (544h5m40.030902559s) [seg_pos={0 0}, upload=477142]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 07:34:40.159906220 (544h5m40.030989265s) [upload=477142]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 07:34:40.159915136 (544h5m40.030998201s) [upload=477142]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 07:34:40.457024204 (544h5m40.328107249s) [upload=477142]: Done uploading segments... Dec 28 07:34:40.457045084 (544h5m40.32812812s) [upload=477142]: Upload finished. err: <nil> Duration: 11.200695833s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 26 out of 60 is failed (threshold=6) [X......X.X..X..XX.XXXXXXX.X...X..X.XX.....X.X.XX.X.X.X..X...] duration: 5.001111986s trace: https://jaeger.dev.storj.io/trace/243a639ca83931a8 FAIL (2025-12-28 07:35:36 UTC) orbiter warning: context deadline exceeded Duration: 5.001138786s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [..XXX.XX.............XX.X...X...X.XXX..X.XX.X.........XX..X.] duration: 5.834307938s trace: https://jaeger.dev.storj.io/trace/133f3e3251246010 FAIL (2025-12-28 07:35:12 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, Piece ID: SHLAKBO2LXCNL34YAV7WE5DBG3OSUEOMNRFGY3DLXLIEBGGZHQCA) context deadline exceeded; piecestore: (Node ID: 12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs, Piece ID: LTQ2GATZ37YL54OTBGYFOYHSTV7GYNZBGVFJ7LLHSHGDVHSTO5IQ) context deadline exceeded ======LOGS====== Dec 28 07:34:37.532705347 (544h5m37.403788382s) [upload=477142]: Starting upload Dec 28 07:34:37.532783416 (544h5m37.403866461s) [upload=477142]: Done starting upload Dec 28 07:34:37.532861695 (544h5m37.40394474s) [upload=477142]: Uploading segments... Dec 28 07:34:37.532979148 (544h5m37.404062183s) [upload=477142]: Got next segment. Inline: false Dec 28 07:34:37.533015638 (544h5m37.404098673s) [upload=477142]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 07:34:37.533020767 (544h5m37.404103812s) [upload=477142]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 07:34:37.778628096 (544h5m37.649711141s) [seg_pos={0 0}, upload=477142]: Begin upload segment... Dec 28 07:34:37.778665327 (544h5m37.649748362s) [seg_pos={0 0}, upload=477142]: Done begin upload segment. Dec 28 07:34:37.778694293 (544h5m37.649777328s) [noise=true , address=145.79.227.14:4839 , piece_id=2SMJMCT2Q4TRJSPBPF37KU5KR6N4W7GTNPM3DOHJH5XFUJGOC3GQ, seg_pos={0 0}, upload=477142]: Uploading piece... Dec 28 07:34:37.779877405 (544h5m37.650960451s) [noise=true , address=145.79.227.39:4014 , piece_id=DTFEKJMIHZYP72HMNCWQUIMJV532PVGB646N3LL7N65OLCLPHXEQ, seg_pos={0 0}, upload=477142]: Uploading piece... Dec 28 07:34:37.780457250 (544h5m37.651540285s) [noise=true , address=145.79.227.44:4194 , piece_id=ORQFOZR4S3QGKNFS3NTLNCY4CN4MB4DUHCQINRGHLMQF72H5D6AA, seg_pos={0 0}, upload=477142]: Uploading piece... Dec 28 07:34:37.780917376 (544h5m37.652000421s) [noise=true , address=145.79.227.48:4301 , piece_id=HHK2NW5UQFHQ2SOGPCE4KNSMYYIFLQKZ5HMGJHMOZDHT6U7DJTHQ, seg_pos={0 0}, upload=477142]: Uploading piece... Dec 28 07:34:37.781384245 (544h5m37.652467291s) [noise=true , address=145.79.227.38:4977 , piece_id=NKZREBCYULHBBQJUEZJOBEO4L2FHMSYBB4L4D5777JE6DVDWOFPQ, seg_pos={0 0}, upload=477142]: Uploading piece... Dec 28 07:34:37.781762274 (544h5m37.652845319s) [noise=true , address=145.79.227.7:4295 , piece_id=2LMCBOGQZNLJFUUDOEKDNSB2JBUTWGVXU5E7XEUSAJN2RKZFNULQ, seg_pos={0 0}, upload=477142]: Uploading piece... Dec 28 07:34:37.782192785 (544h5m37.65327582s) [noise=true , address=45.91.116.4:4939 , piece_id=K4PM7J7FNH7RFN3BN42MYNNS5YXKVTL7QHXIBPG6CLFFTUEGF7EA, seg_pos={0 0}, upload=477142]: Uploading piece... Dec 28 07:34:37.782622242 (544h5m37.653705277s) [noise=true , address=145.79.227.42:4121 , piece_id=JY52UW3TWMFP2X3YWQOWIHWWVLX2TRQXVS3C2TEBLOTDQGK2FMGA, seg_pos={0 0}, upload=477142]: Uploading piece... Dec 28 07:34:37.782986907 (544h5m37.654069942s) [noise=true , address=145.79.227.23:4547 , piece_id=IBUYL6SJ6BK6HPZ6MWXOCDUZX6RGFSQCCWRGLDTFLH4PNFNOVVZA, seg_pos={0 0}, upload=477142]: Uploading piece... Dec 28 07:34:37.783358294 (544h5m37.654441329s) [noise=true , address=145.79.227.16:4909 , piece_id=AHT57DVJN62EZ35M7VOR3Q6WXFILPYRQIX44VICANKC7FYR5IMTQ, seg_pos={0 0}, upload=477142]: Uploading piece... Dec 28 07:34:37.783741303 (544h5m37.654824348s) [noise=true , address=145.79.227.41:4086 , piece_id=D5JRM2DFKV75RLZQNKGIT4W5WXU5L4DDD6YDW3GZF5EUFMNIVF4A, seg_pos={0 0}, upload=477142]: Uploading piece... Dec 28 07:34:37.784137518 (544h5m37.655220563s) [noise=true , address=145.79.227.19:4078 , piece_id=2CFG67OTHVOLV2FLRBN54T5A77RM7XEGMG3TDZUXGBXIICSHJMZA, seg_pos={0 0}, upload=477142]: Uploading piece... Dec 28 07:34:37.784483848 (544h5m37.655566883s) [noise=true , address=45.91.116.8:4045 , piece_id=SHLAKBO2LXCNL34YAV7WE5DBG3OSUEOMNRFGY3DLXLIEBGGZHQCA, seg_pos={0 0}, upload=477142]: Uploading piece... Dec 28 07:34:37.784772717 (544h5m37.655855752s) [noise=true , address=145.79.227.2:4113 , piece_id=CYODBMBDDCCTQJK4CSPCZI43WLMK5D6HCXPUCVPXH4JHRES7XVVA, seg_pos={0 0}, upload=477142]: Uploading piece... Dec 28 07:34:37.785084731 (544h5m37.656167767s) [noise=true , address=45.91.116.9:4263 , piece_id=HFLF2R732JKFUEZPELMGNXFCI7SCNGC3HEHP4ZAK6EIH2BHZRUDQ, seg_pos={0 0}, upload=477142]: Uploading piece... Dec 28 07:34:37.785502818 (544h5m37.656585853s) [noise=true , address=45.91.116.14:4192 , piece_id=LTQ2GATZ37YL54OTBGYFOYHSTV7GYNZBGVFJ7LLHSHGDVHSTO5IQ, seg_pos={0 0}, upload=477142]: Uploading piece... Dec 28 07:34:39.496817391 (544h5m39.367900426s) [upload=477142]: Next returned nil segment Dec 28 07:34:39.496834414 (544h5m39.367917459s) [upload=477142]: Waiting for error group managing segments... Dec 28 07:34:39.782997869 (544h5m39.654080904s) [noise=true , address=145.79.227.38:4977 , piece_id=NKZREBCYULHBBQJUEZJOBEO4L2FHMSYBB4L4D5777JE6DVDWOFPQ, seg_pos={0 0}, upload=477142]: Done uploading piece. err: <nil> Dec 28 07:34:39.788982756 (544h5m39.660065801s) [noise=true , address=145.79.227.14:4839 , piece_id=2SMJMCT2Q4TRJSPBPF37KU5KR6N4W7GTNPM3DOHJH5XFUJGOC3GQ, seg_pos={0 0}, upload=477142]: Done uploading piece. err: <nil> Dec 28 07:34:39.793466995 (544h5m39.66455004s) [noise=true , address=145.79.227.19:4078 , piece_id=2CFG67OTHVOLV2FLRBN54T5A77RM7XEGMG3TDZUXGBXIICSHJMZA, seg_pos={0 0}, upload=477142]: Done uploading piece. err: <nil> Dec 28 07:34:39.794004298 (544h5m39.665087333s) [noise=true , address=145.79.227.41:4086 , piece_id=D5JRM2DFKV75RLZQNKGIT4W5WXU5L4DDD6YDW3GZF5EUFMNIVF4A, seg_pos={0 0}, upload=477142]: Done uploading piece. err: <nil> Dec 28 07:34:39.795054618 (544h5m39.666137663s) [noise=true , address=145.79.227.39:4014 , piece_id=DTFEKJMIHZYP72HMNCWQUIMJV532PVGB646N3LL7N65OLCLPHXEQ, seg_pos={0 0}, upload=477142]: Done uploading piece. err: <nil> Dec 28 07:34:39.799205542 (544h5m39.670288577s) [noise=true , address=145.79.227.7:4295 , piece_id=2LMCBOGQZNLJFUUDOEKDNSB2JBUTWGVXU5E7XEUSAJN2RKZFNULQ, seg_pos={0 0}, upload=477142]: Done uploading piece. err: <nil> Dec 28 07:34:39.799745961 (544h5m39.670828996s) [noise=true , address=145.79.227.2:4113 , piece_id=CYODBMBDDCCTQJK4CSPCZI43WLMK5D6HCXPUCVPXH4JHRES7XVVA, seg_pos={0 0}, upload=477142]: Done uploading piece. err: <nil> Dec 28 07:34:39.810738302 (544h5m39.681821337s) [noise=true , address=145.79.227.48:4301 , piece_id=HHK2NW5UQFHQ2SOGPCE4KNSMYYIFLQKZ5HMGJHMOZDHT6U7DJTHQ, seg_pos={0 0}, upload=477142]: Done uploading piece. err: <nil> Dec 28 07:34:39.814975691 (544h5m39.686058736s) [noise=true , address=145.79.227.23:4547 , piece_id=IBUYL6SJ6BK6HPZ6MWXOCDUZX6RGFSQCCWRGLDTFLH4PNFNOVVZA, seg_pos={0 0}, upload=477142]: Done uploading piece. err: <nil> Dec 28 07:34:39.816049686 (544h5m39.687132731s) [noise=true , address=145.79.227.16:4909 , piece_id=AHT57DVJN62EZ35M7VOR3Q6WXFILPYRQIX44VICANKC7FYR5IMTQ, seg_pos={0 0}, upload=477142]: Done uploading piece. err: <nil> Dec 28 07:34:39.823296435 (544h5m39.69437948s) [noise=true , address=145.79.227.42:4121 , piece_id=JY52UW3TWMFP2X3YWQOWIHWWVLX2TRQXVS3C2TEBLOTDQGK2FMGA, seg_pos={0 0}, upload=477142]: Done uploading piece. err: <nil> Dec 28 07:34:40.133349220 (544h5m40.004432265s) [noise=true , address=45.91.116.14:4192 , piece_id=LTQ2GATZ37YL54OTBGYFOYHSTV7GYNZBGVFJ7LLHSHGDVHSTO5IQ, seg_pos={0 0}, upload=477142]: Done uploading piece. err: <nil> Dec 28 07:34:40.159334831 (544h5m40.030417866s) [noise=true , address=45.91.116.8:4045 , piece_id=SHLAKBO2LXCNL34YAV7WE5DBG3OSUEOMNRFGY3DLXLIEBGGZHQCA, seg_pos={0 0}, upload=477142]: Done uploading piece. err: <nil> Dec 28 07:34:40.159355731 (544h5m40.030438776s) [seg_pos={0 0}, upload=477142]: Segment reached optimal threshold of 13 pieces. Dec 28 07:34:40.159548138 (544h5m40.030631183s) [noise=true , address=45.91.116.4:4939 , piece_id=K4PM7J7FNH7RFN3BN42MYNNS5YXKVTL7QHXIBPG6CLFFTUEGF7EA, seg_pos={0 0}, upload=477142]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: expected piece hash; context canceled; EOF Dec 28 07:34:40.159554710 (544h5m40.030637765s) [noise=true , address=45.91.116.4:4939 , piece_id=K4PM7J7FNH7RFN3BN42MYNNS5YXKVTL7QHXIBPG6CLFFTUEGF7EA, seg_pos={0 0}, upload=477142]: Long tail cancellation due to optimal threshold Dec 28 07:34:40.159709835 (544h5m40.03079287s) [noise=true , address=145.79.227.44:4194 , piece_id=ORQFOZR4S3QGKNFS3NTLNCY4CN4MB4DUHCQINRGHLMQF72H5D6AA, seg_pos={0 0}, upload=477142]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh): protocol: expected piece hash; context canceled; EOF Dec 28 07:34:40.159717720 (544h5m40.030800755s) [noise=true , address=145.79.227.44:4194 , piece_id=ORQFOZR4S3QGKNFS3NTLNCY4CN4MB4DUHCQINRGHLMQF72H5D6AA, seg_pos={0 0}, upload=477142]: Long tail cancellation due to optimal threshold Dec 28 07:34:40.159785700 (544h5m40.030868735s) [noise=true , address=45.91.116.9:4263 , piece_id=HFLF2R732JKFUEZPELMGNXFCI7SCNGC3HEHP4ZAK6EIH2BHZRUDQ, seg_pos={0 0}, upload=477142]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF): protocol: expected piece hash; context canceled; EOF Dec 28 07:34:40.159795018 (544h5m40.030878063s) [noise=true , address=45.91.116.9:4263 , piece_id=HFLF2R732JKFUEZPELMGNXFCI7SCNGC3HEHP4ZAK6EIH2BHZRUDQ, seg_pos={0 0}, upload=477142]: Long tail cancellation due to optimal threshold Dec 28 07:34:40.159819514 (544h5m40.030902559s) [seg_pos={0 0}, upload=477142]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 07:34:40.159906220 (544h5m40.030989265s) [upload=477142]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 07:34:40.159915136 (544h5m40.030998201s) [upload=477142]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 07:34:40.457024204 (544h5m40.328107249s) [upload=477142]: Done uploading segments... Dec 28 07:34:40.457045084 (544h5m40.32812812s) [upload=477142]: Upload finished. err: <nil> Duration: 5.834367691s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [...XXX.XX.............XX.X...X...X.XXX..X.XX.X.........XX..X] duration: 30.003136616s trace: https://jaeger.dev.storj.io/trace/133f3e3251246010 FAIL (2025-12-28 07:35:06 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, Piece ID: SHLAKBO2LXCNL34YAV7WE5DBG3OSUEOMNRFGY3DLXLIEBGGZHQCA) context deadline exceeded; piecestore: (Node ID: 12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs, Piece ID: LTQ2GATZ37YL54OTBGYFOYHSTV7GYNZBGVFJ7LLHSHGDVHSTO5IQ) context deadline exceeded ======LOGS====== Dec 28 07:34:37.532705347 (544h5m37.403788382s) [upload=477142]: Starting upload Dec 28 07:34:37.532783416 (544h5m37.403866461s) [upload=477142]: Done starting upload Dec 28 07:34:37.532861695 (544h5m37.40394474s) [upload=477142]: Uploading segments... Dec 28 07:34:37.532979148 (544h5m37.404062183s) [upload=477142]: Got next segment. Inline: false Dec 28 07:34:37.533015638 (544h5m37.404098673s) [upload=477142]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 07:34:37.533020767 (544h5m37.404103812s) [upload=477142]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 07:34:37.778628096 (544h5m37.649711141s) [seg_pos={0 0}, upload=477142]: Begin upload segment... Dec 28 07:34:37.778665327 (544h5m37.649748362s) [seg_pos={0 0}, upload=477142]: Done begin upload segment. Dec 28 07:34:37.778694293 (544h5m37.649777328s) [noise=true , address=145.79.227.14:4839 , piece_id=2SMJMCT2Q4TRJSPBPF37KU5KR6N4W7GTNPM3DOHJH5XFUJGOC3GQ, seg_pos={0 0}, upload=477142]: Uploading piece... Dec 28 07:34:37.779877405 (544h5m37.650960451s) [noise=true , address=145.79.227.39:4014 , piece_id=DTFEKJMIHZYP72HMNCWQUIMJV532PVGB646N3LL7N65OLCLPHXEQ, seg_pos={0 0}, upload=477142]: Uploading piece... Dec 28 07:34:37.780457250 (544h5m37.651540285s) [noise=true , address=145.79.227.44:4194 , piece_id=ORQFOZR4S3QGKNFS3NTLNCY4CN4MB4DUHCQINRGHLMQF72H5D6AA, seg_pos={0 0}, upload=477142]: Uploading piece... Dec 28 07:34:37.780917376 (544h5m37.652000421s) [noise=true , address=145.79.227.48:4301 , piece_id=HHK2NW5UQFHQ2SOGPCE4KNSMYYIFLQKZ5HMGJHMOZDHT6U7DJTHQ, seg_pos={0 0}, upload=477142]: Uploading piece... Dec 28 07:34:37.781384245 (544h5m37.652467291s) [noise=true , address=145.79.227.38:4977 , piece_id=NKZREBCYULHBBQJUEZJOBEO4L2FHMSYBB4L4D5777JE6DVDWOFPQ, seg_pos={0 0}, upload=477142]: Uploading piece... Dec 28 07:34:37.781762274 (544h5m37.652845319s) [noise=true , address=145.79.227.7:4295 , piece_id=2LMCBOGQZNLJFUUDOEKDNSB2JBUTWGVXU5E7XEUSAJN2RKZFNULQ, seg_pos={0 0}, upload=477142]: Uploading piece... Dec 28 07:34:37.782192785 (544h5m37.65327582s) [noise=true , address=45.91.116.4:4939 , piece_id=K4PM7J7FNH7RFN3BN42MYNNS5YXKVTL7QHXIBPG6CLFFTUEGF7EA, seg_pos={0 0}, upload=477142]: Uploading piece... Dec 28 07:34:37.782622242 (544h5m37.653705277s) [noise=true , address=145.79.227.42:4121 , piece_id=JY52UW3TWMFP2X3YWQOWIHWWVLX2TRQXVS3C2TEBLOTDQGK2FMGA, seg_pos={0 0}, upload=477142]: Uploading piece... Dec 28 07:34:37.782986907 (544h5m37.654069942s) [noise=true , address=145.79.227.23:4547 , piece_id=IBUYL6SJ6BK6HPZ6MWXOCDUZX6RGFSQCCWRGLDTFLH4PNFNOVVZA, seg_pos={0 0}, upload=477142]: Uploading piece... Dec 28 07:34:37.783358294 (544h5m37.654441329s) [noise=true , address=145.79.227.16:4909 , piece_id=AHT57DVJN62EZ35M7VOR3Q6WXFILPYRQIX44VICANKC7FYR5IMTQ, seg_pos={0 0}, upload=477142]: Uploading piece... Dec 28 07:34:37.783741303 (544h5m37.654824348s) [noise=true , address=145.79.227.41:4086 , piece_id=D5JRM2DFKV75RLZQNKGIT4W5WXU5L4DDD6YDW3GZF5EUFMNIVF4A, seg_pos={0 0}, upload=477142]: Uploading piece... Dec 28 07:34:37.784137518 (544h5m37.655220563s) [noise=true , address=145.79.227.19:4078 , piece_id=2CFG67OTHVOLV2FLRBN54T5A77RM7XEGMG3TDZUXGBXIICSHJMZA, seg_pos={0 0}, upload=477142]: Uploading piece... Dec 28 07:34:37.784483848 (544h5m37.655566883s) [noise=true , address=45.91.116.8:4045 , piece_id=SHLAKBO2LXCNL34YAV7WE5DBG3OSUEOMNRFGY3DLXLIEBGGZHQCA, seg_pos={0 0}, upload=477142]: Uploading piece... Dec 28 07:34:37.784772717 (544h5m37.655855752s) [noise=true , address=145.79.227.2:4113 , piece_id=CYODBMBDDCCTQJK4CSPCZI43WLMK5D6HCXPUCVPXH4JHRES7XVVA, seg_pos={0 0}, upload=477142]: Uploading piece... Dec 28 07:34:37.785084731 (544h5m37.656167767s) [noise=true , address=45.91.116.9:4263 , piece_id=HFLF2R732JKFUEZPELMGNXFCI7SCNGC3HEHP4ZAK6EIH2BHZRUDQ, seg_pos={0 0}, upload=477142]: Uploading piece... Dec 28 07:34:37.785502818 (544h5m37.656585853s) [noise=true , address=45.91.116.14:4192 , piece_id=LTQ2GATZ37YL54OTBGYFOYHSTV7GYNZBGVFJ7LLHSHGDVHSTO5IQ, seg_pos={0 0}, upload=477142]: Uploading piece... Dec 28 07:34:39.496817391 (544h5m39.367900426s) [upload=477142]: Next returned nil segment Dec 28 07:34:39.496834414 (544h5m39.367917459s) [upload=477142]: Waiting for error group managing segments... Dec 28 07:34:39.782997869 (544h5m39.654080904s) [noise=true , address=145.79.227.38:4977 , piece_id=NKZREBCYULHBBQJUEZJOBEO4L2FHMSYBB4L4D5777JE6DVDWOFPQ, seg_pos={0 0}, upload=477142]: Done uploading piece. err: <nil> Dec 28 07:34:39.788982756 (544h5m39.660065801s) [noise=true , address=145.79.227.14:4839 , piece_id=2SMJMCT2Q4TRJSPBPF37KU5KR6N4W7GTNPM3DOHJH5XFUJGOC3GQ, seg_pos={0 0}, upload=477142]: Done uploading piece. err: <nil> Dec 28 07:34:39.793466995 (544h5m39.66455004s) [noise=true , address=145.79.227.19:4078 , piece_id=2CFG67OTHVOLV2FLRBN54T5A77RM7XEGMG3TDZUXGBXIICSHJMZA, seg_pos={0 0}, upload=477142]: Done uploading piece. err: <nil> Dec 28 07:34:39.794004298 (544h5m39.665087333s) [noise=true , address=145.79.227.41:4086 , piece_id=D5JRM2DFKV75RLZQNKGIT4W5WXU5L4DDD6YDW3GZF5EUFMNIVF4A, seg_pos={0 0}, upload=477142]: Done uploading piece. err: <nil> Dec 28 07:34:39.795054618 (544h5m39.666137663s) [noise=true , address=145.79.227.39:4014 , piece_id=DTFEKJMIHZYP72HMNCWQUIMJV532PVGB646N3LL7N65OLCLPHXEQ, seg_pos={0 0}, upload=477142]: Done uploading piece. err: <nil> Dec 28 07:34:39.799205542 (544h5m39.670288577s) [noise=true , address=145.79.227.7:4295 , piece_id=2LMCBOGQZNLJFUUDOEKDNSB2JBUTWGVXU5E7XEUSAJN2RKZFNULQ, seg_pos={0 0}, upload=477142]: Done uploading piece. err: <nil> Dec 28 07:34:39.799745961 (544h5m39.670828996s) [noise=true , address=145.79.227.2:4113 , piece_id=CYODBMBDDCCTQJK4CSPCZI43WLMK5D6HCXPUCVPXH4JHRES7XVVA, seg_pos={0 0}, upload=477142]: Done uploading piece. err: <nil> Dec 28 07:34:39.810738302 (544h5m39.681821337s) [noise=true , address=145.79.227.48:4301 , piece_id=HHK2NW5UQFHQ2SOGPCE4KNSMYYIFLQKZ5HMGJHMOZDHT6U7DJTHQ, seg_pos={0 0}, upload=477142]: Done uploading piece. err: <nil> Dec 28 07:34:39.814975691 (544h5m39.686058736s) [noise=true , address=145.79.227.23:4547 , piece_id=IBUYL6SJ6BK6HPZ6MWXOCDUZX6RGFSQCCWRGLDTFLH4PNFNOVVZA, seg_pos={0 0}, upload=477142]: Done uploading piece. err: <nil> Dec 28 07:34:39.816049686 (544h5m39.687132731s) [noise=true , address=145.79.227.16:4909 , piece_id=AHT57DVJN62EZ35M7VOR3Q6WXFILPYRQIX44VICANKC7FYR5IMTQ, seg_pos={0 0}, upload=477142]: Done uploading piece. err: <nil> Dec 28 07:34:39.823296435 (544h5m39.69437948s) [noise=true , address=145.79.227.42:4121 , piece_id=JY52UW3TWMFP2X3YWQOWIHWWVLX2TRQXVS3C2TEBLOTDQGK2FMGA, seg_pos={0 0}, upload=477142]: Done uploading piece. err: <nil> Dec 28 07:34:40.133349220 (544h5m40.004432265s) [noise=true , address=45.91.116.14:4192 , piece_id=LTQ2GATZ37YL54OTBGYFOYHSTV7GYNZBGVFJ7LLHSHGDVHSTO5IQ, seg_pos={0 0}, upload=477142]: Done uploading piece. err: <nil> Dec 28 07:34:40.159334831 (544h5m40.030417866s) [noise=true , address=45.91.116.8:4045 , piece_id=SHLAKBO2LXCNL34YAV7WE5DBG3OSUEOMNRFGY3DLXLIEBGGZHQCA, seg_pos={0 0}, upload=477142]: Done uploading piece. err: <nil> Dec 28 07:34:40.159355731 (544h5m40.030438776s) [seg_pos={0 0}, upload=477142]: Segment reached optimal threshold of 13 pieces. Dec 28 07:34:40.159548138 (544h5m40.030631183s) [noise=true , address=45.91.116.4:4939 , piece_id=K4PM7J7FNH7RFN3BN42MYNNS5YXKVTL7QHXIBPG6CLFFTUEGF7EA, seg_pos={0 0}, upload=477142]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: expected piece hash; context canceled; EOF Dec 28 07:34:40.159554710 (544h5m40.030637765s) [noise=true , address=45.91.116.4:4939 , piece_id=K4PM7J7FNH7RFN3BN42MYNNS5YXKVTL7QHXIBPG6CLFFTUEGF7EA, seg_pos={0 0}, upload=477142]: Long tail cancellation due to optimal threshold Dec 28 07:34:40.159709835 (544h5m40.03079287s) [noise=true , address=145.79.227.44:4194 , piece_id=ORQFOZR4S3QGKNFS3NTLNCY4CN4MB4DUHCQINRGHLMQF72H5D6AA, seg_pos={0 0}, upload=477142]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh): protocol: expected piece hash; context canceled; EOF Dec 28 07:34:40.159717720 (544h5m40.030800755s) [noise=true , address=145.79.227.44:4194 , piece_id=ORQFOZR4S3QGKNFS3NTLNCY4CN4MB4DUHCQINRGHLMQF72H5D6AA, seg_pos={0 0}, upload=477142]: Long tail cancellation due to optimal threshold Dec 28 07:34:40.159785700 (544h5m40.030868735s) [noise=true , address=45.91.116.9:4263 , piece_id=HFLF2R732JKFUEZPELMGNXFCI7SCNGC3HEHP4ZAK6EIH2BHZRUDQ, seg_pos={0 0}, upload=477142]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF): protocol: expected piece hash; context canceled; EOF Dec 28 07:34:40.159795018 (544h5m40.030878063s) [noise=true , address=45.91.116.9:4263 , piece_id=HFLF2R732JKFUEZPELMGNXFCI7SCNGC3HEHP4ZAK6EIH2BHZRUDQ, seg_pos={0 0}, upload=477142]: Long tail cancellation due to optimal threshold Dec 28 07:34:40.159819514 (544h5m40.030902559s) [seg_pos={0 0}, upload=477142]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 07:34:40.159906220 (544h5m40.030989265s) [upload=477142]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 07:34:40.159915136 (544h5m40.030998201s) [upload=477142]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 07:34:40.457024204 (544h5m40.328107249s) [upload=477142]: Done uploading segments... Dec 28 07:34:40.457045084 (544h5m40.32812812s) [upload=477142]: Upload finished. err: <nil> Duration: 30.00316538s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 27 out of 60 is failed (threshold=6) [XX......X.X..X..XX.XXXXXXX.X...X..X.XX.....X.X.XX.X.X.X..X..] duration: 5.000870396s trace: https://jaeger.dev.storj.io/trace/243a639ca83931a8 FAIL (2025-12-28 07:34:44 UTC) orbiter warning: context deadline exceeded Duration: 5.00089387s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [X...XXX.XX.............XX.X...X...X.XXX..X.XX.X.........XX..] duration: 6.172457413s trace: https://jaeger.dev.storj.io/trace/473623a4904d4261 FAIL (2025-12-28 07:34:02 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 28 07:32:44.240497757 (544h3m44.111580792s) [upload=477113]: Starting upload Dec 28 07:32:44.240588450 (544h3m44.111671485s) [upload=477113]: Done starting upload Dec 28 07:32:44.240634848 (544h3m44.111717893s) [upload=477113]: Uploading segments... Dec 28 07:32:44.240770406 (544h3m44.111853441s) [upload=477113]: Got next segment. Inline: false Dec 28 07:32:44.240806224 (544h3m44.111889269s) [upload=477113]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 07:32:44.240810602 (544h3m44.111893637s) [upload=477113]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 07:32:44.487624329 (544h3m44.358707374s) [seg_pos={0 0}, upload=477113]: Begin upload segment... Dec 28 07:32:44.487683422 (544h3m44.358766467s) [seg_pos={0 0}, upload=477113]: Done begin upload segment. Dec 28 07:32:44.487709241 (544h3m44.358792277s) [noise=true , address=145.79.227.48:4302 , piece_id=XBFPNOBIMYAI3SY3DX6NJ32MLV2V3EY5RO5NJWDC2DB74HAGV4OA, seg_pos={0 0}, upload=477113]: Uploading piece... Dec 28 07:32:44.487728167 (544h3m44.358811202s) [noise=true , address=45.91.116.11:4117 , piece_id=R7ITWQHE74YQ4ZZPAF446WMZGEEOWWXPWARIBNHBWVJS5TPZCUAQ, seg_pos={0 0}, upload=477113]: Uploading piece... Dec 28 07:32:44.488226196 (544h3m44.359309231s) [noise=true , address=45.91.116.8:4048 , piece_id=I3XMASBS6ZTDZ534PV5ACIAFRHXXAVEOJX3FY5R66I2ILWFLICJQ, seg_pos={0 0}, upload=477113]: Uploading piece... Dec 28 07:32:44.488643430 (544h3m44.359726476s) [noise=true , address=145.79.227.49:4339 , piece_id=YV4UIJF6MIWKUPNK5G5HNX3G3ZC2CRYCQMN4WLIGATAFKIKS7UMQ, seg_pos={0 0}, upload=477113]: Uploading piece... Dec 28 07:32:44.489047169 (544h3m44.360130204s) [noise=true , address=145.79.227.51:4409 , piece_id=LREJSZUDP6FZJGM6AAZJEA7V76OUC42H5MGUEJQQUNRDMZURSILQ, seg_pos={0 0}, upload=477113]: Uploading piece... Dec 28 07:32:44.489431901 (544h3m44.360514946s) [noise=true , address=45.91.116.9:4264 , piece_id=T5DEYGCYYCANLRSELJ3UP655BWWXWFI2RQU6IB53CQQNJNWL54KQ, seg_pos={0 0}, upload=477113]: Uploading piece... Dec 28 07:32:44.489883050 (544h3m44.360966095s) [noise=true , address=145.79.227.2:4114 , piece_id=NGY7MMJSZGRYMQ3WDWA3L275BUL2SL3MCEC2ZDQZTRIFNY3BEXWQ, seg_pos={0 0}, upload=477113]: Uploading piece... Dec 28 07:32:44.490421075 (544h3m44.36150412s) [noise=true , address=145.79.227.42:4123 , piece_id=2XHW5FI2LBQ2APZAEFTR3YQKEBHFNTKBS3LAWDEFWDRDVUUUHXPA, seg_pos={0 0}, upload=477113]: Uploading piece... Dec 28 07:32:44.490914324 (544h3m44.361997359s) [noise=true , address=45.91.116.5:4297 , piece_id=5A4XAOVNP54HYVAERN4TQCIIXI5BBFBLC2N6UOQ2VDEO4M63HC6Q, seg_pos={0 0}, upload=477113]: Uploading piece... Dec 28 07:32:44.492126693 (544h3m44.363209738s) [noise=true , address=145.79.227.37:4567 , piece_id=3URDEEVRYKV3BIGW6A73IL4M245G4KOSKUHNFIPEBUIM3F4XPSGQ, seg_pos={0 0}, upload=477113]: Uploading piece... Dec 28 07:32:44.492466810 (544h3m44.363549845s) [noise=true , address=145.79.227.52:4447 , piece_id=5PLWTF6JZPRU2G5CWHNHN7JSH4UHGJ7TSMHBBTXI3QXIHZRQDHMQ, seg_pos={0 0}, upload=477113]: Uploading piece... Dec 28 07:32:44.493431968 (544h3m44.364515013s) [noise=true , address=45.91.116.6:4974 , piece_id=MP36B4NV3MZORC7XIFM3RUQQ3IV4IWPUEN74Y6G7TE56BLH77CVA, seg_pos={0 0}, upload=477113]: Uploading piece... Dec 28 07:32:44.494011893 (544h3m44.365094928s) [noise=true , address=77.90.31.14:4661 , piece_id=YSOAWULTF726GNS2P4ZF726OLHQZII7JQWJHQG2W5KEVYNJ3LVXA, seg_pos={0 0}, upload=477113]: Uploading piece... Dec 28 07:32:44.495210184 (544h3m44.366293219s) [noise=true , address=145.79.227.4:4187 , piece_id=EJS5O6DGBRF62552RNN5VWYOI4RWLH3M6K457SCPD2G3XWRUHL3A, seg_pos={0 0}, upload=477113]: Uploading piece... Dec 28 07:32:44.495510416 (544h3m44.366593451s) [noise=true , address=145.79.227.27:4691 , piece_id=UUVX6U2E6WJRZLLEAV2Q5TMU3WYZV5Q2CKG77KTT42YUZMLPLZSQ, seg_pos={0 0}, upload=477113]: Uploading piece... Dec 28 07:32:44.497052302 (544h3m44.368135347s) [noise=true , address=145.79.227.50:4374 , piece_id=RPNP2ZCELJB6LVCMB5LEANVN2RL65CE3J2PJQPH64423GXCZH7CQ, seg_pos={0 0}, upload=477113]: Uploading piece... Dec 28 07:32:46.209008177 (544h3m46.080091222s) [upload=477113]: Next returned nil segment Dec 28 07:32:46.209022685 (544h3m46.08010573s) [upload=477113]: Waiting for error group managing segments... Dec 28 07:32:46.496400843 (544h3m46.367483878s) [noise=true , address=145.79.227.51:4409 , piece_id=LREJSZUDP6FZJGM6AAZJEA7V76OUC42H5MGUEJQQUNRDMZURSILQ, seg_pos={0 0}, upload=477113]: Done uploading piece. err: <nil> Dec 28 07:32:46.504113970 (544h3m46.375197015s) [noise=true , address=145.79.227.49:4339 , piece_id=YV4UIJF6MIWKUPNK5G5HNX3G3ZC2CRYCQMN4WLIGATAFKIKS7UMQ, seg_pos={0 0}, upload=477113]: Done uploading piece. err: <nil> Dec 28 07:32:46.506725583 (544h3m46.377808628s) [noise=true , address=145.79.227.52:4447 , piece_id=5PLWTF6JZPRU2G5CWHNHN7JSH4UHGJ7TSMHBBTXI3QXIHZRQDHMQ, seg_pos={0 0}, upload=477113]: Done uploading piece. err: <nil> Dec 28 07:32:46.506776109 (544h3m46.377859154s) [noise=true , address=145.79.227.50:4374 , piece_id=RPNP2ZCELJB6LVCMB5LEANVN2RL65CE3J2PJQPH64423GXCZH7CQ, seg_pos={0 0}, upload=477113]: Done uploading piece. err: <nil> Dec 28 07:32:46.507358879 (544h3m46.378441914s) [noise=true , address=145.79.227.37:4567 , piece_id=3URDEEVRYKV3BIGW6A73IL4M245G4KOSKUHNFIPEBUIM3F4XPSGQ, seg_pos={0 0}, upload=477113]: Done uploading piece. err: <nil> Dec 28 07:32:46.508111683 (544h3m46.379194718s) [noise=true , address=145.79.227.48:4302 , piece_id=XBFPNOBIMYAI3SY3DX6NJ32MLV2V3EY5RO5NJWDC2DB74HAGV4OA, seg_pos={0 0}, upload=477113]: Done uploading piece. err: <nil> Dec 28 07:32:46.512570683 (544h3m46.383653719s) [noise=true , address=145.79.227.4:4187 , piece_id=EJS5O6DGBRF62552RNN5VWYOI4RWLH3M6K457SCPD2G3XWRUHL3A, seg_pos={0 0}, upload=477113]: Done uploading piece. err: <nil> Dec 28 07:32:46.517092935 (544h3m46.38817597s) [noise=true , address=145.79.227.27:4691 , piece_id=UUVX6U2E6WJRZLLEAV2Q5TMU3WYZV5Q2CKG77KTT42YUZMLPLZSQ, seg_pos={0 0}, upload=477113]: Done uploading piece. err: <nil> Dec 28 07:32:46.519382214 (544h3m46.390465259s) [noise=true , address=145.79.227.2:4114 , piece_id=NGY7MMJSZGRYMQ3WDWA3L275BUL2SL3MCEC2ZDQZTRIFNY3BEXWQ, seg_pos={0 0}, upload=477113]: Done uploading piece. err: <nil> Dec 28 07:32:46.525905726 (544h3m46.396988761s) [noise=true , address=145.79.227.42:4123 , piece_id=2XHW5FI2LBQ2APZAEFTR3YQKEBHFNTKBS3LAWDEFWDRDVUUUHXPA, seg_pos={0 0}, upload=477113]: Done uploading piece. err: <nil> Dec 28 07:32:46.683211985 (544h3m46.55429503s) [noise=true , address=77.90.31.14:4661 , piece_id=YSOAWULTF726GNS2P4ZF726OLHQZII7JQWJHQG2W5KEVYNJ3LVXA, seg_pos={0 0}, upload=477113]: Done uploading piece. err: <nil> Dec 28 07:32:46.765759259 (544h3m46.636842304s) [noise=true , address=45.91.116.5:4297 , piece_id=5A4XAOVNP54HYVAERN4TQCIIXI5BBFBLC2N6UOQ2VDEO4M63HC6Q, seg_pos={0 0}, upload=477113]: Done uploading piece. err: <nil> Dec 28 07:32:46.889565127 (544h3m46.760648162s) [noise=true , address=45.91.116.9:4264 , piece_id=T5DEYGCYYCANLRSELJ3UP655BWWXWFI2RQU6IB53CQQNJNWL54KQ, seg_pos={0 0}, upload=477113]: Done uploading piece. err: <nil> Dec 28 07:32:46.889588843 (544h3m46.760671878s) [seg_pos={0 0}, upload=477113]: Segment reached optimal threshold of 13 pieces. Dec 28 07:32:46.889917027 (544h3m46.761000072s) [noise=true , address=45.91.116.6:4974 , piece_id=MP36B4NV3MZORC7XIFM3RUQQ3IV4IWPUEN74Y6G7TE56BLH77CVA, seg_pos={0 0}, upload=477113]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: expected piece hash; context canceled; EOF Dec 28 07:32:46.890004975 (544h3m46.76108802s) [noise=true , address=45.91.116.6:4974 , piece_id=MP36B4NV3MZORC7XIFM3RUQQ3IV4IWPUEN74Y6G7TE56BLH77CVA, seg_pos={0 0}, upload=477113]: Long tail cancellation due to optimal threshold Dec 28 07:32:46.890190097 (544h3m46.761273142s) [noise=true , address=45.91.116.11:4117 , piece_id=R7ITWQHE74YQ4ZZPAF446WMZGEEOWWXPWARIBNHBWVJS5TPZCUAQ, seg_pos={0 0}, upload=477113]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6): protocol: write tcp 188.166.225.170:32998->45.91.116.11:4117: use of closed network connection; write tcp 188.166.225.170:32998->45.91.116.11:4117: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:32998->45.91.116.11:4117: use of closed network connection Dec 28 07:32:46.890205767 (544h3m46.761288802s) [noise=true , address=45.91.116.11:4117 , piece_id=R7ITWQHE74YQ4ZZPAF446WMZGEEOWWXPWARIBNHBWVJS5TPZCUAQ, seg_pos={0 0}, upload=477113]: Long tail cancellation due to optimal threshold Dec 28 07:32:46.890287393 (544h3m46.761370448s) [noise=true , address=45.91.116.8:4048 , piece_id=I3XMASBS6ZTDZ534PV5ACIAFRHXXAVEOJX3FY5R66I2ILWFLICJQ, seg_pos={0 0}, upload=477113]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: expected piece hash; context canceled; EOF Dec 28 07:32:46.890297672 (544h3m46.761380717s) [noise=true , address=45.91.116.8:4048 , piece_id=I3XMASBS6ZTDZ534PV5ACIAFRHXXAVEOJX3FY5R66I2ILWFLICJQ, seg_pos={0 0}, upload=477113]: Long tail cancellation due to optimal threshold Dec 28 07:32:46.890319694 (544h3m46.761402739s) [seg_pos={0 0}, upload=477113]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 07:32:46.890418853 (544h3m46.761501898s) [upload=477113]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 07:32:46.890426608 (544h3m46.761509663s) [upload=477113]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 07:32:47.188962392 (544h3m47.060045438s) [upload=477113]: Done uploading segments... Dec 28 07:32:47.188983352 (544h3m47.060066387s) [upload=477113]: Upload finished. err: <nil> Duration: 6.172488453s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [.X...XXX.XX.............XX.X...X...X.XXX..X.XX.X.........XX.] duration: 6.003619757s trace: https://jaeger.dev.storj.io/trace/473623a4904d4261 FAIL (2025-12-28 07:33:42 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 28 07:32:44.240497757 (544h3m44.111580792s) [upload=477113]: Starting upload Dec 28 07:32:44.240588450 (544h3m44.111671485s) [upload=477113]: Done starting upload Dec 28 07:32:44.240634848 (544h3m44.111717893s) [upload=477113]: Uploading segments... Dec 28 07:32:44.240770406 (544h3m44.111853441s) [upload=477113]: Got next segment. Inline: false Dec 28 07:32:44.240806224 (544h3m44.111889269s) [upload=477113]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 07:32:44.240810602 (544h3m44.111893637s) [upload=477113]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 07:32:44.487624329 (544h3m44.358707374s) [seg_pos={0 0}, upload=477113]: Begin upload segment... Dec 28 07:32:44.487683422 (544h3m44.358766467s) [seg_pos={0 0}, upload=477113]: Done begin upload segment. Dec 28 07:32:44.487709241 (544h3m44.358792277s) [noise=true , address=145.79.227.48:4302 , piece_id=XBFPNOBIMYAI3SY3DX6NJ32MLV2V3EY5RO5NJWDC2DB74HAGV4OA, seg_pos={0 0}, upload=477113]: Uploading piece... Dec 28 07:32:44.487728167 (544h3m44.358811202s) [noise=true , address=45.91.116.11:4117 , piece_id=R7ITWQHE74YQ4ZZPAF446WMZGEEOWWXPWARIBNHBWVJS5TPZCUAQ, seg_pos={0 0}, upload=477113]: Uploading piece... Dec 28 07:32:44.488226196 (544h3m44.359309231s) [noise=true , address=45.91.116.8:4048 , piece_id=I3XMASBS6ZTDZ534PV5ACIAFRHXXAVEOJX3FY5R66I2ILWFLICJQ, seg_pos={0 0}, upload=477113]: Uploading piece... Dec 28 07:32:44.488643430 (544h3m44.359726476s) [noise=true , address=145.79.227.49:4339 , piece_id=YV4UIJF6MIWKUPNK5G5HNX3G3ZC2CRYCQMN4WLIGATAFKIKS7UMQ, seg_pos={0 0}, upload=477113]: Uploading piece... Dec 28 07:32:44.489047169 (544h3m44.360130204s) [noise=true , address=145.79.227.51:4409 , piece_id=LREJSZUDP6FZJGM6AAZJEA7V76OUC42H5MGUEJQQUNRDMZURSILQ, seg_pos={0 0}, upload=477113]: Uploading piece... Dec 28 07:32:44.489431901 (544h3m44.360514946s) [noise=true , address=45.91.116.9:4264 , piece_id=T5DEYGCYYCANLRSELJ3UP655BWWXWFI2RQU6IB53CQQNJNWL54KQ, seg_pos={0 0}, upload=477113]: Uploading piece... Dec 28 07:32:44.489883050 (544h3m44.360966095s) [noise=true , address=145.79.227.2:4114 , piece_id=NGY7MMJSZGRYMQ3WDWA3L275BUL2SL3MCEC2ZDQZTRIFNY3BEXWQ, seg_pos={0 0}, upload=477113]: Uploading piece... Dec 28 07:32:44.490421075 (544h3m44.36150412s) [noise=true , address=145.79.227.42:4123 , piece_id=2XHW5FI2LBQ2APZAEFTR3YQKEBHFNTKBS3LAWDEFWDRDVUUUHXPA, seg_pos={0 0}, upload=477113]: Uploading piece... Dec 28 07:32:44.490914324 (544h3m44.361997359s) [noise=true , address=45.91.116.5:4297 , piece_id=5A4XAOVNP54HYVAERN4TQCIIXI5BBFBLC2N6UOQ2VDEO4M63HC6Q, seg_pos={0 0}, upload=477113]: Uploading piece... Dec 28 07:32:44.492126693 (544h3m44.363209738s) [noise=true , address=145.79.227.37:4567 , piece_id=3URDEEVRYKV3BIGW6A73IL4M245G4KOSKUHNFIPEBUIM3F4XPSGQ, seg_pos={0 0}, upload=477113]: Uploading piece... Dec 28 07:32:44.492466810 (544h3m44.363549845s) [noise=true , address=145.79.227.52:4447 , piece_id=5PLWTF6JZPRU2G5CWHNHN7JSH4UHGJ7TSMHBBTXI3QXIHZRQDHMQ, seg_pos={0 0}, upload=477113]: Uploading piece... Dec 28 07:32:44.493431968 (544h3m44.364515013s) [noise=true , address=45.91.116.6:4974 , piece_id=MP36B4NV3MZORC7XIFM3RUQQ3IV4IWPUEN74Y6G7TE56BLH77CVA, seg_pos={0 0}, upload=477113]: Uploading piece... Dec 28 07:32:44.494011893 (544h3m44.365094928s) [noise=true , address=77.90.31.14:4661 , piece_id=YSOAWULTF726GNS2P4ZF726OLHQZII7JQWJHQG2W5KEVYNJ3LVXA, seg_pos={0 0}, upload=477113]: Uploading piece... Dec 28 07:32:44.495210184 (544h3m44.366293219s) [noise=true , address=145.79.227.4:4187 , piece_id=EJS5O6DGBRF62552RNN5VWYOI4RWLH3M6K457SCPD2G3XWRUHL3A, seg_pos={0 0}, upload=477113]: Uploading piece... Dec 28 07:32:44.495510416 (544h3m44.366593451s) [noise=true , address=145.79.227.27:4691 , piece_id=UUVX6U2E6WJRZLLEAV2Q5TMU3WYZV5Q2CKG77KTT42YUZMLPLZSQ, seg_pos={0 0}, upload=477113]: Uploading piece... Dec 28 07:32:44.497052302 (544h3m44.368135347s) [noise=true , address=145.79.227.50:4374 , piece_id=RPNP2ZCELJB6LVCMB5LEANVN2RL65CE3J2PJQPH64423GXCZH7CQ, seg_pos={0 0}, upload=477113]: Uploading piece... Dec 28 07:32:46.209008177 (544h3m46.080091222s) [upload=477113]: Next returned nil segment Dec 28 07:32:46.209022685 (544h3m46.08010573s) [upload=477113]: Waiting for error group managing segments... Dec 28 07:32:46.496400843 (544h3m46.367483878s) [noise=true , address=145.79.227.51:4409 , piece_id=LREJSZUDP6FZJGM6AAZJEA7V76OUC42H5MGUEJQQUNRDMZURSILQ, seg_pos={0 0}, upload=477113]: Done uploading piece. err: <nil> Dec 28 07:32:46.504113970 (544h3m46.375197015s) [noise=true , address=145.79.227.49:4339 , piece_id=YV4UIJF6MIWKUPNK5G5HNX3G3ZC2CRYCQMN4WLIGATAFKIKS7UMQ, seg_pos={0 0}, upload=477113]: Done uploading piece. err: <nil> Dec 28 07:32:46.506725583 (544h3m46.377808628s) [noise=true , address=145.79.227.52:4447 , piece_id=5PLWTF6JZPRU2G5CWHNHN7JSH4UHGJ7TSMHBBTXI3QXIHZRQDHMQ, seg_pos={0 0}, upload=477113]: Done uploading piece. err: <nil> Dec 28 07:32:46.506776109 (544h3m46.377859154s) [noise=true , address=145.79.227.50:4374 , piece_id=RPNP2ZCELJB6LVCMB5LEANVN2RL65CE3J2PJQPH64423GXCZH7CQ, seg_pos={0 0}, upload=477113]: Done uploading piece. err: <nil> Dec 28 07:32:46.507358879 (544h3m46.378441914s) [noise=true , address=145.79.227.37:4567 , piece_id=3URDEEVRYKV3BIGW6A73IL4M245G4KOSKUHNFIPEBUIM3F4XPSGQ, seg_pos={0 0}, upload=477113]: Done uploading piece. err: <nil> Dec 28 07:32:46.508111683 (544h3m46.379194718s) [noise=true , address=145.79.227.48:4302 , piece_id=XBFPNOBIMYAI3SY3DX6NJ32MLV2V3EY5RO5NJWDC2DB74HAGV4OA, seg_pos={0 0}, upload=477113]: Done uploading piece. err: <nil> Dec 28 07:32:46.512570683 (544h3m46.383653719s) [noise=true , address=145.79.227.4:4187 , piece_id=EJS5O6DGBRF62552RNN5VWYOI4RWLH3M6K457SCPD2G3XWRUHL3A, seg_pos={0 0}, upload=477113]: Done uploading piece. err: <nil> Dec 28 07:32:46.517092935 (544h3m46.38817597s) [noise=true , address=145.79.227.27:4691 , piece_id=UUVX6U2E6WJRZLLEAV2Q5TMU3WYZV5Q2CKG77KTT42YUZMLPLZSQ, seg_pos={0 0}, upload=477113]: Done uploading piece. err: <nil> Dec 28 07:32:46.519382214 (544h3m46.390465259s) [noise=true , address=145.79.227.2:4114 , piece_id=NGY7MMJSZGRYMQ3WDWA3L275BUL2SL3MCEC2ZDQZTRIFNY3BEXWQ, seg_pos={0 0}, upload=477113]: Done uploading piece. err: <nil> Dec 28 07:32:46.525905726 (544h3m46.396988761s) [noise=true , address=145.79.227.42:4123 , piece_id=2XHW5FI2LBQ2APZAEFTR3YQKEBHFNTKBS3LAWDEFWDRDVUUUHXPA, seg_pos={0 0}, upload=477113]: Done uploading piece. err: <nil> Dec 28 07:32:46.683211985 (544h3m46.55429503s) [noise=true , address=77.90.31.14:4661 , piece_id=YSOAWULTF726GNS2P4ZF726OLHQZII7JQWJHQG2W5KEVYNJ3LVXA, seg_pos={0 0}, upload=477113]: Done uploading piece. err: <nil> Dec 28 07:32:46.765759259 (544h3m46.636842304s) [noise=true , address=45.91.116.5:4297 , piece_id=5A4XAOVNP54HYVAERN4TQCIIXI5BBFBLC2N6UOQ2VDEO4M63HC6Q, seg_pos={0 0}, upload=477113]: Done uploading piece. err: <nil> Dec 28 07:32:46.889565127 (544h3m46.760648162s) [noise=true , address=45.91.116.9:4264 , piece_id=T5DEYGCYYCANLRSELJ3UP655BWWXWFI2RQU6IB53CQQNJNWL54KQ, seg_pos={0 0}, upload=477113]: Done uploading piece. err: <nil> Dec 28 07:32:46.889588843 (544h3m46.760671878s) [seg_pos={0 0}, upload=477113]: Segment reached optimal threshold of 13 pieces. Dec 28 07:32:46.889917027 (544h3m46.761000072s) [noise=true , address=45.91.116.6:4974 , piece_id=MP36B4NV3MZORC7XIFM3RUQQ3IV4IWPUEN74Y6G7TE56BLH77CVA, seg_pos={0 0}, upload=477113]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: expected piece hash; context canceled; EOF Dec 28 07:32:46.890004975 (544h3m46.76108802s) [noise=true , address=45.91.116.6:4974 , piece_id=MP36B4NV3MZORC7XIFM3RUQQ3IV4IWPUEN74Y6G7TE56BLH77CVA, seg_pos={0 0}, upload=477113]: Long tail cancellation due to optimal threshold Dec 28 07:32:46.890190097 (544h3m46.761273142s) [noise=true , address=45.91.116.11:4117 , piece_id=R7ITWQHE74YQ4ZZPAF446WMZGEEOWWXPWARIBNHBWVJS5TPZCUAQ, seg_pos={0 0}, upload=477113]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6): protocol: write tcp 188.166.225.170:32998->45.91.116.11:4117: use of closed network connection; write tcp 188.166.225.170:32998->45.91.116.11:4117: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:32998->45.91.116.11:4117: use of closed network connection Dec 28 07:32:46.890205767 (544h3m46.761288802s) [noise=true , address=45.91.116.11:4117 , piece_id=R7ITWQHE74YQ4ZZPAF446WMZGEEOWWXPWARIBNHBWVJS5TPZCUAQ, seg_pos={0 0}, upload=477113]: Long tail cancellation due to optimal threshold Dec 28 07:32:46.890287393 (544h3m46.761370448s) [noise=true , address=45.91.116.8:4048 , piece_id=I3XMASBS6ZTDZ534PV5ACIAFRHXXAVEOJX3FY5R66I2ILWFLICJQ, seg_pos={0 0}, upload=477113]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: expected piece hash; context canceled; EOF Dec 28 07:32:46.890297672 (544h3m46.761380717s) [noise=true , address=45.91.116.8:4048 , piece_id=I3XMASBS6ZTDZ534PV5ACIAFRHXXAVEOJX3FY5R66I2ILWFLICJQ, seg_pos={0 0}, upload=477113]: Long tail cancellation due to optimal threshold Dec 28 07:32:46.890319694 (544h3m46.761402739s) [seg_pos={0 0}, upload=477113]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 07:32:46.890418853 (544h3m46.761501898s) [upload=477113]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 07:32:46.890426608 (544h3m46.761509663s) [upload=477113]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 07:32:47.188962392 (544h3m47.060045438s) [upload=477113]: Done uploading segments... Dec 28 07:32:47.188983352 (544h3m47.060066387s) [upload=477113]: Upload finished. err: <nil> Duration: 6.003648581s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 27 out of 60 is failed (threshold=6) [.XX......X.X..X..XX.XXXXXXX.X...X..X.XX.....X.X.XX.X.X.X..X.] duration: 5.001326195s trace: https://jaeger.dev.storj.io/trace/243a639ca83931a8 FAIL (2025-12-28 07:33:38 UTC) orbiter warning: context deadline exceeded Duration: 5.001350791s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [..X...XXX.XX.............XX.X...X...X.XXX..X.XX.X.........XX] duration: 30.003002141s trace: https://jaeger.dev.storj.io/trace/473623a4904d4261 FAIL (2025-12-28 07:33:13 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 28 07:32:44.240497757 (544h3m44.111580792s) [upload=477113]: Starting upload Dec 28 07:32:44.240588450 (544h3m44.111671485s) [upload=477113]: Done starting upload Dec 28 07:32:44.240634848 (544h3m44.111717893s) [upload=477113]: Uploading segments... Dec 28 07:32:44.240770406 (544h3m44.111853441s) [upload=477113]: Got next segment. Inline: false Dec 28 07:32:44.240806224 (544h3m44.111889269s) [upload=477113]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 07:32:44.240810602 (544h3m44.111893637s) [upload=477113]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 07:32:44.487624329 (544h3m44.358707374s) [seg_pos={0 0}, upload=477113]: Begin upload segment... Dec 28 07:32:44.487683422 (544h3m44.358766467s) [seg_pos={0 0}, upload=477113]: Done begin upload segment. Dec 28 07:32:44.487709241 (544h3m44.358792277s) [noise=true , address=145.79.227.48:4302 , piece_id=XBFPNOBIMYAI3SY3DX6NJ32MLV2V3EY5RO5NJWDC2DB74HAGV4OA, seg_pos={0 0}, upload=477113]: Uploading piece... Dec 28 07:32:44.487728167 (544h3m44.358811202s) [noise=true , address=45.91.116.11:4117 , piece_id=R7ITWQHE74YQ4ZZPAF446WMZGEEOWWXPWARIBNHBWVJS5TPZCUAQ, seg_pos={0 0}, upload=477113]: Uploading piece... Dec 28 07:32:44.488226196 (544h3m44.359309231s) [noise=true , address=45.91.116.8:4048 , piece_id=I3XMASBS6ZTDZ534PV5ACIAFRHXXAVEOJX3FY5R66I2ILWFLICJQ, seg_pos={0 0}, upload=477113]: Uploading piece... Dec 28 07:32:44.488643430 (544h3m44.359726476s) [noise=true , address=145.79.227.49:4339 , piece_id=YV4UIJF6MIWKUPNK5G5HNX3G3ZC2CRYCQMN4WLIGATAFKIKS7UMQ, seg_pos={0 0}, upload=477113]: Uploading piece... Dec 28 07:32:44.489047169 (544h3m44.360130204s) [noise=true , address=145.79.227.51:4409 , piece_id=LREJSZUDP6FZJGM6AAZJEA7V76OUC42H5MGUEJQQUNRDMZURSILQ, seg_pos={0 0}, upload=477113]: Uploading piece... Dec 28 07:32:44.489431901 (544h3m44.360514946s) [noise=true , address=45.91.116.9:4264 , piece_id=T5DEYGCYYCANLRSELJ3UP655BWWXWFI2RQU6IB53CQQNJNWL54KQ, seg_pos={0 0}, upload=477113]: Uploading piece... Dec 28 07:32:44.489883050 (544h3m44.360966095s) [noise=true , address=145.79.227.2:4114 , piece_id=NGY7MMJSZGRYMQ3WDWA3L275BUL2SL3MCEC2ZDQZTRIFNY3BEXWQ, seg_pos={0 0}, upload=477113]: Uploading piece... Dec 28 07:32:44.490421075 (544h3m44.36150412s) [noise=true , address=145.79.227.42:4123 , piece_id=2XHW5FI2LBQ2APZAEFTR3YQKEBHFNTKBS3LAWDEFWDRDVUUUHXPA, seg_pos={0 0}, upload=477113]: Uploading piece... Dec 28 07:32:44.490914324 (544h3m44.361997359s) [noise=true , address=45.91.116.5:4297 , piece_id=5A4XAOVNP54HYVAERN4TQCIIXI5BBFBLC2N6UOQ2VDEO4M63HC6Q, seg_pos={0 0}, upload=477113]: Uploading piece... Dec 28 07:32:44.492126693 (544h3m44.363209738s) [noise=true , address=145.79.227.37:4567 , piece_id=3URDEEVRYKV3BIGW6A73IL4M245G4KOSKUHNFIPEBUIM3F4XPSGQ, seg_pos={0 0}, upload=477113]: Uploading piece... Dec 28 07:32:44.492466810 (544h3m44.363549845s) [noise=true , address=145.79.227.52:4447 , piece_id=5PLWTF6JZPRU2G5CWHNHN7JSH4UHGJ7TSMHBBTXI3QXIHZRQDHMQ, seg_pos={0 0}, upload=477113]: Uploading piece... Dec 28 07:32:44.493431968 (544h3m44.364515013s) [noise=true , address=45.91.116.6:4974 , piece_id=MP36B4NV3MZORC7XIFM3RUQQ3IV4IWPUEN74Y6G7TE56BLH77CVA, seg_pos={0 0}, upload=477113]: Uploading piece... Dec 28 07:32:44.494011893 (544h3m44.365094928s) [noise=true , address=77.90.31.14:4661 , piece_id=YSOAWULTF726GNS2P4ZF726OLHQZII7JQWJHQG2W5KEVYNJ3LVXA, seg_pos={0 0}, upload=477113]: Uploading piece... Dec 28 07:32:44.495210184 (544h3m44.366293219s) [noise=true , address=145.79.227.4:4187 , piece_id=EJS5O6DGBRF62552RNN5VWYOI4RWLH3M6K457SCPD2G3XWRUHL3A, seg_pos={0 0}, upload=477113]: Uploading piece... Dec 28 07:32:44.495510416 (544h3m44.366593451s) [noise=true , address=145.79.227.27:4691 , piece_id=UUVX6U2E6WJRZLLEAV2Q5TMU3WYZV5Q2CKG77KTT42YUZMLPLZSQ, seg_pos={0 0}, upload=477113]: Uploading piece... Dec 28 07:32:44.497052302 (544h3m44.368135347s) [noise=true , address=145.79.227.50:4374 , piece_id=RPNP2ZCELJB6LVCMB5LEANVN2RL65CE3J2PJQPH64423GXCZH7CQ, seg_pos={0 0}, upload=477113]: Uploading piece... Dec 28 07:32:46.209008177 (544h3m46.080091222s) [upload=477113]: Next returned nil segment Dec 28 07:32:46.209022685 (544h3m46.08010573s) [upload=477113]: Waiting for error group managing segments... Dec 28 07:32:46.496400843 (544h3m46.367483878s) [noise=true , address=145.79.227.51:4409 , piece_id=LREJSZUDP6FZJGM6AAZJEA7V76OUC42H5MGUEJQQUNRDMZURSILQ, seg_pos={0 0}, upload=477113]: Done uploading piece. err: <nil> Dec 28 07:32:46.504113970 (544h3m46.375197015s) [noise=true , address=145.79.227.49:4339 , piece_id=YV4UIJF6MIWKUPNK5G5HNX3G3ZC2CRYCQMN4WLIGATAFKIKS7UMQ, seg_pos={0 0}, upload=477113]: Done uploading piece. err: <nil> Dec 28 07:32:46.506725583 (544h3m46.377808628s) [noise=true , address=145.79.227.52:4447 , piece_id=5PLWTF6JZPRU2G5CWHNHN7JSH4UHGJ7TSMHBBTXI3QXIHZRQDHMQ, seg_pos={0 0}, upload=477113]: Done uploading piece. err: <nil> Dec 28 07:32:46.506776109 (544h3m46.377859154s) [noise=true , address=145.79.227.50:4374 , piece_id=RPNP2ZCELJB6LVCMB5LEANVN2RL65CE3J2PJQPH64423GXCZH7CQ, seg_pos={0 0}, upload=477113]: Done uploading piece. err: <nil> Dec 28 07:32:46.507358879 (544h3m46.378441914s) [noise=true , address=145.79.227.37:4567 , piece_id=3URDEEVRYKV3BIGW6A73IL4M245G4KOSKUHNFIPEBUIM3F4XPSGQ, seg_pos={0 0}, upload=477113]: Done uploading piece. err: <nil> Dec 28 07:32:46.508111683 (544h3m46.379194718s) [noise=true , address=145.79.227.48:4302 , piece_id=XBFPNOBIMYAI3SY3DX6NJ32MLV2V3EY5RO5NJWDC2DB74HAGV4OA, seg_pos={0 0}, upload=477113]: Done uploading piece. err: <nil> Dec 28 07:32:46.512570683 (544h3m46.383653719s) [noise=true , address=145.79.227.4:4187 , piece_id=EJS5O6DGBRF62552RNN5VWYOI4RWLH3M6K457SCPD2G3XWRUHL3A, seg_pos={0 0}, upload=477113]: Done uploading piece. err: <nil> Dec 28 07:32:46.517092935 (544h3m46.38817597s) [noise=true , address=145.79.227.27:4691 , piece_id=UUVX6U2E6WJRZLLEAV2Q5TMU3WYZV5Q2CKG77KTT42YUZMLPLZSQ, seg_pos={0 0}, upload=477113]: Done uploading piece. err: <nil> Dec 28 07:32:46.519382214 (544h3m46.390465259s) [noise=true , address=145.79.227.2:4114 , piece_id=NGY7MMJSZGRYMQ3WDWA3L275BUL2SL3MCEC2ZDQZTRIFNY3BEXWQ, seg_pos={0 0}, upload=477113]: Done uploading piece. err: <nil> Dec 28 07:32:46.525905726 (544h3m46.396988761s) [noise=true , address=145.79.227.42:4123 , piece_id=2XHW5FI2LBQ2APZAEFTR3YQKEBHFNTKBS3LAWDEFWDRDVUUUHXPA, seg_pos={0 0}, upload=477113]: Done uploading piece. err: <nil> Dec 28 07:32:46.683211985 (544h3m46.55429503s) [noise=true , address=77.90.31.14:4661 , piece_id=YSOAWULTF726GNS2P4ZF726OLHQZII7JQWJHQG2W5KEVYNJ3LVXA, seg_pos={0 0}, upload=477113]: Done uploading piece. err: <nil> Dec 28 07:32:46.765759259 (544h3m46.636842304s) [noise=true , address=45.91.116.5:4297 , piece_id=5A4XAOVNP54HYVAERN4TQCIIXI5BBFBLC2N6UOQ2VDEO4M63HC6Q, seg_pos={0 0}, upload=477113]: Done uploading piece. err: <nil> Dec 28 07:32:46.889565127 (544h3m46.760648162s) [noise=true , address=45.91.116.9:4264 , piece_id=T5DEYGCYYCANLRSELJ3UP655BWWXWFI2RQU6IB53CQQNJNWL54KQ, seg_pos={0 0}, upload=477113]: Done uploading piece. err: <nil> Dec 28 07:32:46.889588843 (544h3m46.760671878s) [seg_pos={0 0}, upload=477113]: Segment reached optimal threshold of 13 pieces. Dec 28 07:32:46.889917027 (544h3m46.761000072s) [noise=true , address=45.91.116.6:4974 , piece_id=MP36B4NV3MZORC7XIFM3RUQQ3IV4IWPUEN74Y6G7TE56BLH77CVA, seg_pos={0 0}, upload=477113]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: expected piece hash; context canceled; EOF Dec 28 07:32:46.890004975 (544h3m46.76108802s) [noise=true , address=45.91.116.6:4974 , piece_id=MP36B4NV3MZORC7XIFM3RUQQ3IV4IWPUEN74Y6G7TE56BLH77CVA, seg_pos={0 0}, upload=477113]: Long tail cancellation due to optimal threshold Dec 28 07:32:46.890190097 (544h3m46.761273142s) [noise=true , address=45.91.116.11:4117 , piece_id=R7ITWQHE74YQ4ZZPAF446WMZGEEOWWXPWARIBNHBWVJS5TPZCUAQ, seg_pos={0 0}, upload=477113]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6): protocol: write tcp 188.166.225.170:32998->45.91.116.11:4117: use of closed network connection; write tcp 188.166.225.170:32998->45.91.116.11:4117: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:32998->45.91.116.11:4117: use of closed network connection Dec 28 07:32:46.890205767 (544h3m46.761288802s) [noise=true , address=45.91.116.11:4117 , piece_id=R7ITWQHE74YQ4ZZPAF446WMZGEEOWWXPWARIBNHBWVJS5TPZCUAQ, seg_pos={0 0}, upload=477113]: Long tail cancellation due to optimal threshold Dec 28 07:32:46.890287393 (544h3m46.761370448s) [noise=true , address=45.91.116.8:4048 , piece_id=I3XMASBS6ZTDZ534PV5ACIAFRHXXAVEOJX3FY5R66I2ILWFLICJQ, seg_pos={0 0}, upload=477113]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: expected piece hash; context canceled; EOF Dec 28 07:32:46.890297672 (544h3m46.761380717s) [noise=true , address=45.91.116.8:4048 , piece_id=I3XMASBS6ZTDZ534PV5ACIAFRHXXAVEOJX3FY5R66I2ILWFLICJQ, seg_pos={0 0}, upload=477113]: Long tail cancellation due to optimal threshold Dec 28 07:32:46.890319694 (544h3m46.761402739s) [seg_pos={0 0}, upload=477113]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 07:32:46.890418853 (544h3m46.761501898s) [upload=477113]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 07:32:46.890426608 (544h3m46.761509663s) [upload=477113]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 07:32:47.188962392 (544h3m47.060045438s) [upload=477113]: Done uploading segments... Dec 28 07:32:47.188983352 (544h3m47.060066387s) [upload=477113]: Upload finished. err: <nil> Duration: 30.003024283s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [...X...XXX.XX.............XX.X...X...X.XXX..X.XX.X.........X] duration: 30.002656424s trace: https://jaeger.dev.storj.io/trace/767fecaa0e0c2e50 FAIL (2025-12-28 07:32:29 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 28 07:31:59.990380153 (544h2m59.861463188s) [upload=477101]: Starting upload Dec 28 07:31:59.990467479 (544h2m59.861550524s) [upload=477101]: Done starting upload Dec 28 07:31:59.990559795 (544h2m59.86164284s) [upload=477101]: Uploading segments... Dec 28 07:31:59.990769364 (544h2m59.861852409s) [upload=477101]: Got next segment. Inline: false Dec 28 07:31:59.990851109 (544h2m59.861934144s) [upload=477101]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 07:31:59.990858434 (544h2m59.861941469s) [upload=477101]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 07:32:00.262927470 (544h3m0.134010516s) [seg_pos={0 0}, upload=477101]: Begin upload segment... Dec 28 07:32:00.263005018 (544h3m0.134088063s) [seg_pos={0 0}, upload=477101]: Done begin upload segment. Dec 28 07:32:00.263063970 (544h3m0.134147016s) [noise=true , address=45.91.116.10:4081 , piece_id=LQYRUGEFRGMMWFHXA7PERF67MXXEUUGHDQFF77IB2FUXUIEAKRJQ, seg_pos={0 0}, upload=477101]: Uploading piece... Dec 28 07:32:00.263140105 (544h3m0.13422315s) [noise=true , address=145.79.227.4:4185 , piece_id=SKD4YU3AZDVBFR2O3TAHCBYV6CESIV7LROPEMM6LAKQ5GZLE23SA, seg_pos={0 0}, upload=477101]: Uploading piece... Dec 28 07:32:00.263908448 (544h3m0.134991493s) [noise=true , address=77.90.31.10:4520 , piece_id=LFNI742XNIHZVGWEEORHBZTRFQU4L7E26QFNYJHIDEJYF5S55KYQ, seg_pos={0 0}, upload=477101]: Uploading piece... Dec 28 07:32:00.264028306 (544h3m0.135111351s) [noise=true , address=45.91.116.8:4045 , piece_id=25ERRYR5N4AK74BQP26UTZDXUP4HKT6HLHELO72D76S3U32CTHTQ, seg_pos={0 0}, upload=477101]: Uploading piece... Dec 28 07:32:00.264358265 (544h3m0.1354413s) [noise=true , address=145.79.227.17:4972 , piece_id=DPYUI6KY7F6A2BYN3Y7SC3UA7I5PSHD3KARW7L2T6EYKSXG7WUPA, seg_pos={0 0}, upload=477101]: Uploading piece... Dec 28 07:32:00.264588854 (544h3m0.135671899s) [noise=true , address=145.79.227.27:4691 , piece_id=3VXAJZSM7GXQDS2UT4JWMQPGA5S32JCEGH4NXZJQ5PG67S2MBWKA, seg_pos={0 0}, upload=477101]: Uploading piece... Dec 28 07:32:00.264771522 (544h3m0.135854567s) [noise=true , address=145.79.227.46:4602 , piece_id=LUF54AMNN65QDFZIAZVZ5MJ36W5HU3W47OE2HNRBVONTYFK2Z7BA, seg_pos={0 0}, upload=477101]: Uploading piece... Dec 28 07:32:00.265215087 (544h3m0.136298132s) [noise=true , address=145.79.227.7:4295 , piece_id=DQGGIVUWU7J7FNG3NSL5GJ5TO5LHHWT74DJ27GBNAL66VPBLCVXQ, seg_pos={0 0}, upload=477101]: Uploading piece... Dec 28 07:32:00.265440986 (544h3m0.136524032s) [noise=true , address=45.91.116.7:4009 , piece_id=YMHSYC6AM5GGXVKAC5GE3V2RVVI7SPMQCQXA6YWS53AZ3KHFIQAQ, seg_pos={0 0}, upload=477101]: Uploading piece... Dec 28 07:32:00.265851578 (544h3m0.136934624s) [noise=true , address=145.79.227.32:4836 , piece_id=7BINLHG62N6IGY5PVCKV7OQCO6CMIWA4WNHLFIIDY6SZNEK2NOGA, seg_pos={0 0}, upload=477101]: Uploading piece... Dec 28 07:32:00.266055527 (544h3m0.137138572s) [noise=true , address=145.79.227.36:4518 , piece_id=5YHYDULSP5BQ7L27H7R5AUMRAHJZTQAIBJBNTSLORZG4IZ6VOBNA, seg_pos={0 0}, upload=477101]: Uploading piece... Dec 28 07:32:00.266503610 (544h3m0.137586655s) [noise=true , address=145.79.227.16:4910 , piece_id=K2XDNVEZLZE6YSN535F6GEBW4P6DIWUWQ5CRXZDZ6ZM2HKXGGW2Q, seg_pos={0 0}, upload=477101]: Uploading piece... Dec 28 07:32:00.266573764 (544h3m0.137656809s) [noise=true , address=45.91.116.11:4118 , piece_id=AQUCZJD7ZQTVTE3WZ7QNM7VO2AIDLCXCA4V6VIZR5PXCTNNV7JBA, seg_pos={0 0}, upload=477101]: Uploading piece... Dec 28 07:32:00.266930303 (544h3m0.138013338s) [noise=true , address=77.90.31.17:4770 , piece_id=AMVK3VAI6YZZNMELPMNHJORD72K24UVGONDXBWDOI3SHJXYSHZKA, seg_pos={0 0}, upload=477101]: Uploading piece... Dec 28 07:32:00.267180178 (544h3m0.138263213s) [noise=true , address=145.79.227.14:4838 , piece_id=RX6RGW3YPSQH3YZP55DEC6Y64SSLMMAA2HSS2INVTQH4JPIF3JTQ, seg_pos={0 0}, upload=477101]: Uploading piece... Dec 28 07:32:00.268016581 (544h3m0.139099626s) [noise=true , address=145.79.227.42:4121 , piece_id=W4UHAZN3BJJUK7GOBW6TXB7K7DNGAJOO7V5SOYZ7OSJSHOL3OH7Q, seg_pos={0 0}, upload=477101]: Uploading piece... Dec 28 07:32:02.012595326 (544h3m1.883678371s) [upload=477101]: Next returned nil segment Dec 28 07:32:02.012609142 (544h3m1.883692187s) [upload=477101]: Waiting for error group managing segments... Dec 28 07:32:02.304045519 (544h3m2.175128554s) [noise=true , address=145.79.227.46:4602 , piece_id=LUF54AMNN65QDFZIAZVZ5MJ36W5HU3W47OE2HNRBVONTYFK2Z7BA, seg_pos={0 0}, upload=477101]: Done uploading piece. err: <nil> Dec 28 07:32:02.324753360 (544h3m2.195836395s) [noise=true , address=145.79.227.4:4185 , piece_id=SKD4YU3AZDVBFR2O3TAHCBYV6CESIV7LROPEMM6LAKQ5GZLE23SA, seg_pos={0 0}, upload=477101]: Done uploading piece. err: <nil> Dec 28 07:32:02.335026051 (544h3m2.206109096s) [noise=true , address=145.79.227.17:4972 , piece_id=DPYUI6KY7F6A2BYN3Y7SC3UA7I5PSHD3KARW7L2T6EYKSXG7WUPA, seg_pos={0 0}, upload=477101]: Done uploading piece. err: <nil> Dec 28 07:32:02.379183945 (544h3m2.25026699s) [noise=true , address=145.79.227.27:4691 , piece_id=3VXAJZSM7GXQDS2UT4JWMQPGA5S32JCEGH4NXZJQ5PG67S2MBWKA, seg_pos={0 0}, upload=477101]: Done uploading piece. err: <nil> Dec 28 07:32:02.382770554 (544h3m2.2538536s) [noise=true , address=145.79.227.42:4121 , piece_id=W4UHAZN3BJJUK7GOBW6TXB7K7DNGAJOO7V5SOYZ7OSJSHOL3OH7Q, seg_pos={0 0}, upload=477101]: Done uploading piece. err: <nil> Dec 28 07:32:02.393495026 (544h3m2.264578071s) [noise=true , address=145.79.227.7:4295 , piece_id=DQGGIVUWU7J7FNG3NSL5GJ5TO5LHHWT74DJ27GBNAL66VPBLCVXQ, seg_pos={0 0}, upload=477101]: Done uploading piece. err: <nil> Dec 28 07:32:02.394461406 (544h3m2.265544441s) [noise=true , address=145.79.227.36:4518 , piece_id=5YHYDULSP5BQ7L27H7R5AUMRAHJZTQAIBJBNTSLORZG4IZ6VOBNA, seg_pos={0 0}, upload=477101]: Done uploading piece. err: <nil> Dec 28 07:32:02.396350393 (544h3m2.267433438s) [noise=true , address=145.79.227.16:4910 , piece_id=K2XDNVEZLZE6YSN535F6GEBW4P6DIWUWQ5CRXZDZ6ZM2HKXGGW2Q, seg_pos={0 0}, upload=477101]: Done uploading piece. err: <nil> Dec 28 07:32:02.400466241 (544h3m2.271549286s) [noise=true , address=145.79.227.32:4836 , piece_id=7BINLHG62N6IGY5PVCKV7OQCO6CMIWA4WNHLFIIDY6SZNEK2NOGA, seg_pos={0 0}, upload=477101]: Done uploading piece. err: <nil> Dec 28 07:32:02.402175796 (544h3m2.273258841s) [noise=true , address=145.79.227.14:4838 , piece_id=RX6RGW3YPSQH3YZP55DEC6Y64SSLMMAA2HSS2INVTQH4JPIF3JTQ, seg_pos={0 0}, upload=477101]: Done uploading piece. err: <nil> Dec 28 07:32:02.509452149 (544h3m2.380535194s) [noise=true , address=45.91.116.11:4118 , piece_id=AQUCZJD7ZQTVTE3WZ7QNM7VO2AIDLCXCA4V6VIZR5PXCTNNV7JBA, seg_pos={0 0}, upload=477101]: Done uploading piece. err: <nil> Dec 28 07:32:02.556161933 (544h3m2.427244978s) [noise=true , address=77.90.31.10:4520 , piece_id=LFNI742XNIHZVGWEEORHBZTRFQU4L7E26QFNYJHIDEJYF5S55KYQ, seg_pos={0 0}, upload=477101]: Done uploading piece. err: <nil> Dec 28 07:32:02.599843951 (544h3m2.470926996s) [noise=true , address=45.91.116.8:4045 , piece_id=25ERRYR5N4AK74BQP26UTZDXUP4HKT6HLHELO72D76S3U32CTHTQ, seg_pos={0 0}, upload=477101]: Done uploading piece. err: <nil> Dec 28 07:32:02.599871943 (544h3m2.470954988s) [seg_pos={0 0}, upload=477101]: Segment reached optimal threshold of 13 pieces. Dec 28 07:32:02.600167647 (544h3m2.471250682s) [noise=true , address=45.91.116.7:4009 , piece_id=YMHSYC6AM5GGXVKAC5GE3V2RVVI7SPMQCQXA6YWS53AZ3KHFIQAQ, seg_pos={0 0}, upload=477101]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: expected piece hash; context canceled; EOF Dec 28 07:32:02.600174911 (544h3m2.471257946s) [noise=true , address=45.91.116.7:4009 , piece_id=YMHSYC6AM5GGXVKAC5GE3V2RVVI7SPMQCQXA6YWS53AZ3KHFIQAQ, seg_pos={0 0}, upload=477101]: Long tail cancellation due to optimal threshold Dec 28 07:32:02.600354422 (544h3m2.471437457s) [noise=true , address=45.91.116.10:4081 , piece_id=LQYRUGEFRGMMWFHXA7PERF67MXXEUUGHDQFF77IB2FUXUIEAKRJQ, seg_pos={0 0}, upload=477101]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG): protocol: expected piece hash; context canceled; EOF Dec 28 07:32:02.600384981 (544h3m2.471468016s) [noise=true , address=45.91.116.10:4081 , piece_id=LQYRUGEFRGMMWFHXA7PERF67MXXEUUGHDQFF77IB2FUXUIEAKRJQ, seg_pos={0 0}, upload=477101]: Long tail cancellation due to optimal threshold Dec 28 07:32:02.600436740 (544h3m2.471519775s) [noise=true , address=77.90.31.17:4770 , piece_id=AMVK3VAI6YZZNMELPMNHJORD72K24UVGONDXBWDOI3SHJXYSHZKA, seg_pos={0 0}, upload=477101]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R): protocol: write tcp 188.166.225.170:41826->77.90.31.17:4770: use of closed network connection; write tcp 188.166.225.170:41826->77.90.31.17:4770: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41826->77.90.31.17:4770: use of closed network connection Dec 28 07:32:02.600444244 (544h3m2.471527289s) [noise=true , address=77.90.31.17:4770 , piece_id=AMVK3VAI6YZZNMELPMNHJORD72K24UVGONDXBWDOI3SHJXYSHZKA, seg_pos={0 0}, upload=477101]: Long tail cancellation due to optimal threshold Dec 28 07:32:02.600505811 (544h3m2.471588856s) [seg_pos={0 0}, upload=477101]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 07:32:02.600616842 (544h3m2.471699877s) [upload=477101]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 07:32:02.600623234 (544h3m2.471706269s) [upload=477101]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 07:32:02.912361198 (544h3m2.783444263s) [upload=477101]: Done uploading segments... Dec 28 07:32:02.912386396 (544h3m2.783469441s) [upload=477101]: Upload finished. err: <nil> Duration: 30.002678787s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 27 out of 60 is failed (threshold=6) [..XX......X.X..X..XX.XXXXXXX.X...X..X.XX.....X.X.XX.X.X.X..X] duration: 5.001750363s trace: https://jaeger.dev.storj.io/trace/243a639ca83931a8 FAIL (2025-12-28 07:32:26 UTC) orbiter warning: context deadline exceeded Duration: 5.001763447s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 26 out of 60 is failed (threshold=6) [...XX......X.X..X..XX.XXXXXXX.X...X..X.XX.....X.X.XX.X.X.X..] duration: 3.409105357s trace: https://jaeger.dev.storj.io/trace/6bee2b84b91d624b FAIL (2025-12-28 07:31:35 UTC) orbiter warning: context deadline exceeded Duration: 3.40913354s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [....X...XXX.XX.............XX.X...X...X.XXX..X.XX.X.........] duration: 5.764451179s trace: https://jaeger.dev.storj.io/trace/19034703e3293256 FAIL (2025-12-28 07:31:05 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 28 07:24:31.869041095 (543h55m31.740124131s) [upload=476992]: Starting upload Dec 28 07:24:31.869091793 (543h55m31.740174838s) [upload=476992]: Done starting upload Dec 28 07:24:31.869203846 (543h55m31.740286901s) [upload=476992]: Uploading segments... Dec 28 07:24:31.869341798 (543h55m31.740424854s) [upload=476992]: Got next segment. Inline: false Dec 28 07:24:31.869387505 (543h55m31.74047054s) [upload=476992]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 07:24:31.869392474 (543h55m31.74047551s) [upload=476992]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 07:24:32.112124481 (543h55m31.983207526s) [seg_pos={0 0}, upload=476992]: Begin upload segment... Dec 28 07:24:32.112187041 (543h55m31.983270086s) [seg_pos={0 0}, upload=476992]: Done begin upload segment. Dec 28 07:24:32.112262514 (543h55m31.983345549s) [noise=true , address=145.79.227.45:4231 , piece_id=M4CCTNOTHYYE2YXIGKONI26OH7W2I4R3IQJH2TXY6ZFKLRYTBPNA, seg_pos={0 0}, upload=476992]: Uploading piece... Dec 28 07:24:32.112347206 (543h55m31.983430251s) [noise=true , address=145.79.227.31:4798 , piece_id=6MA7SUPGRYZ7RZSIURS3IFEE4B7VRDFWLYVTWCQFHSYMGX5ULP6A, seg_pos={0 0}, upload=476992]: Uploading piece... Dec 28 07:24:32.112868147 (543h55m31.983951192s) [noise=true , address=145.79.227.7:4293 , piece_id=ILHS7U65IXGQUHGUZOSKDLR75D2ADT6VMY2DF3Q44JGA6NC35YQA, seg_pos={0 0}, upload=476992]: Uploading piece... Dec 28 07:24:32.113275393 (543h55m31.984358438s) [noise=true , address=145.79.227.17:4972 , piece_id=2DEF4NOZF5IEFEHLS3C245RYLN6TKKHIEYAVXJWK2EZDTDB5LZ6A, seg_pos={0 0}, upload=476992]: Uploading piece... Dec 28 07:24:32.114689235 (543h55m31.98577228s) [noise=true , address=45.91.116.5:4297 , piece_id=TRXYL52PWBQI3U3H5S7XFNEVTIKZXJXVUL7B5H3YL7RWQNVAWSDA, seg_pos={0 0}, upload=476992]: Uploading piece... Dec 28 07:24:32.115243370 (543h55m31.986326416s) [noise=true , address=145.79.227.35:4481 , piece_id=45CJ7I6U6OJLKJNNCECWBTRWKL3MDXPAP5TSWR57JIXAQ7QTF5CQ, seg_pos={0 0}, upload=476992]: Uploading piece... Dec 28 07:24:32.115709027 (543h55m31.986792072s) [noise=true , address=45.91.116.8:4046 , piece_id=2FECWK73M4LDD3WHQTQY53KETVJY4FHMSDABWBLOLODMZIJZVNFQ, seg_pos={0 0}, upload=476992]: Uploading piece... Dec 28 07:24:32.116664757 (543h55m31.987747792s) [noise=true , address=45.91.116.12:4225 , piece_id=FEEGNTEUKADXJA6Q525AUFVFNTDAWYGP5HFEVNVJN6Q4CQN2BB3A, seg_pos={0 0}, upload=476992]: Uploading piece... Dec 28 07:24:32.117199886 (543h55m31.988282931s) [noise=true , address=145.79.227.14:4839 , piece_id=HWBXXJKYTV7C6TOBWMK7U5QJVRROMVPLZLPAP7URZ2E6J6TKBOVA, seg_pos={0 0}, upload=476992]: Uploading piece... Dec 28 07:24:32.118394961 (543h55m31.989477996s) [noise=true , address=145.79.227.37:4567 , piece_id=OPWBMOJSFIVAV2ZQVTSPIUU3RWB6R35COP75KWQW7N4EM3EUE3MA, seg_pos={0 0}, upload=476992]: Uploading piece... Dec 28 07:24:32.119110775 (543h55m31.99019382s) [noise=true , address=77.90.31.13:4626 , piece_id=7DPB5UBH7ABVQEHEZAJRVY5LTLSFU3D5DIZH2NZLYAKUEHZ4UHDA, seg_pos={0 0}, upload=476992]: Uploading piece... Dec 28 07:24:32.122486352 (543h55m31.993569397s) [noise=true , address=45.91.116.13:4156 , piece_id=R7RVEBAT5VGQZHUHRLEPBS7STP7XVA6AY4YSLPRIXJKUVKCFF4VQ, seg_pos={0 0}, upload=476992]: Uploading piece... Dec 28 07:24:32.124638781 (543h55m31.995721816s) [noise=true , address=145.79.227.16:4911 , piece_id=HDA2REQUIC2Z7XRKP77UDNOWFVITNSXSMNFQA7NHIVYJNBB6SIZQ, seg_pos={0 0}, upload=476992]: Uploading piece... Dec 28 07:24:32.124849412 (543h55m31.995932457s) [noise=true , address=145.79.227.18:4007 , piece_id=TZ6KLDXOBKVHULQB3E4WLXRACWPTA6C5JZFWITYFFSXK2D5KOCXA, seg_pos={0 0}, upload=476992]: Uploading piece... Dec 28 07:24:32.126406407 (543h55m31.997489442s) [noise=true , address=45.91.116.10:4084 , piece_id=HVGRMOQXOHAFUNUXY2J42H5OJL6XDEMLO3BLROQEFJ6INUAK4I4A, seg_pos={0 0}, upload=476992]: Uploading piece... Dec 28 07:24:32.127679831 (543h55m31.998762886s) [noise=true , address=145.79.227.42:4123 , piece_id=TIHM4TDHHEXOITN4UL6ZTQUGPD3KU67QNH5Y2HZX7IYQ6EBHKXUQ, seg_pos={0 0}, upload=476992]: Uploading piece... Dec 28 07:24:33.749682045 (543h55m33.62076508s) [upload=476992]: Next returned nil segment Dec 28 07:24:33.749699568 (543h55m33.620782613s) [upload=476992]: Waiting for error group managing segments... Dec 28 07:24:34.119458798 (543h55m33.990541843s) [noise=true , address=145.79.227.45:4231 , piece_id=M4CCTNOTHYYE2YXIGKONI26OH7W2I4R3IQJH2TXY6ZFKLRYTBPNA, seg_pos={0 0}, upload=476992]: Done uploading piece. err: <nil> Dec 28 07:24:34.121601728 (543h55m33.992684763s) [noise=true , address=145.79.227.17:4972 , piece_id=2DEF4NOZF5IEFEHLS3C245RYLN6TKKHIEYAVXJWK2EZDTDB5LZ6A, seg_pos={0 0}, upload=476992]: Done uploading piece. err: <nil> Dec 28 07:24:34.128325031 (543h55m33.999408066s) [noise=true , address=145.79.227.37:4567 , piece_id=OPWBMOJSFIVAV2ZQVTSPIUU3RWB6R35COP75KWQW7N4EM3EUE3MA, seg_pos={0 0}, upload=476992]: Done uploading piece. err: <nil> Dec 28 07:24:34.128912158 (543h55m33.999995203s) [noise=true , address=145.79.227.16:4911 , piece_id=HDA2REQUIC2Z7XRKP77UDNOWFVITNSXSMNFQA7NHIVYJNBB6SIZQ, seg_pos={0 0}, upload=476992]: Done uploading piece. err: <nil> Dec 28 07:24:34.131137215 (543h55m34.00222025s) [noise=true , address=145.79.227.42:4123 , piece_id=TIHM4TDHHEXOITN4UL6ZTQUGPD3KU67QNH5Y2HZX7IYQ6EBHKXUQ, seg_pos={0 0}, upload=476992]: Done uploading piece. err: <nil> Dec 28 07:24:34.133603162 (543h55m34.004686197s) [noise=true , address=145.79.227.7:4293 , piece_id=ILHS7U65IXGQUHGUZOSKDLR75D2ADT6VMY2DF3Q44JGA6NC35YQA, seg_pos={0 0}, upload=476992]: Done uploading piece. err: <nil> Dec 28 07:24:34.140904865 (543h55m34.0119879s) [noise=true , address=145.79.227.35:4481 , piece_id=45CJ7I6U6OJLKJNNCECWBTRWKL3MDXPAP5TSWR57JIXAQ7QTF5CQ, seg_pos={0 0}, upload=476992]: Done uploading piece. err: <nil> Dec 28 07:24:34.146817494 (543h55m34.017900539s) [noise=true , address=145.79.227.14:4839 , piece_id=HWBXXJKYTV7C6TOBWMK7U5QJVRROMVPLZLPAP7URZ2E6J6TKBOVA, seg_pos={0 0}, upload=476992]: Done uploading piece. err: <nil> Dec 28 07:24:34.146901404 (543h55m34.017984449s) [noise=true , address=145.79.227.31:4798 , piece_id=6MA7SUPGRYZ7RZSIURS3IFEE4B7VRDFWLYVTWCQFHSYMGX5ULP6A, seg_pos={0 0}, upload=476992]: Done uploading piece. err: <nil> Dec 28 07:24:34.154751622 (543h55m34.025834657s) [noise=true , address=145.79.227.18:4007 , piece_id=TZ6KLDXOBKVHULQB3E4WLXRACWPTA6C5JZFWITYFFSXK2D5KOCXA, seg_pos={0 0}, upload=476992]: Done uploading piece. err: <nil> Dec 28 07:24:34.362215111 (543h55m34.233298146s) [noise=true , address=45.91.116.12:4225 , piece_id=FEEGNTEUKADXJA6Q525AUFVFNTDAWYGP5HFEVNVJN6Q4CQN2BB3A, seg_pos={0 0}, upload=476992]: Done uploading piece. err: <nil> Dec 28 07:24:34.372017326 (543h55m34.243100371s) [noise=true , address=45.91.116.13:4156 , piece_id=R7RVEBAT5VGQZHUHRLEPBS7STP7XVA6AY4YSLPRIXJKUVKCFF4VQ, seg_pos={0 0}, upload=476992]: Done uploading piece. err: <nil> Dec 28 07:24:34.565728377 (543h55m34.436811422s) [noise=true , address=45.91.116.10:4084 , piece_id=HVGRMOQXOHAFUNUXY2J42H5OJL6XDEMLO3BLROQEFJ6INUAK4I4A, seg_pos={0 0}, upload=476992]: Done uploading piece. err: <nil> Dec 28 07:24:34.565747725 (543h55m34.43683076s) [seg_pos={0 0}, upload=476992]: Segment reached optimal threshold of 13 pieces. Dec 28 07:24:34.566162044 (543h55m34.437245089s) [noise=true , address=77.90.31.13:4626 , piece_id=7DPB5UBH7ABVQEHEZAJRVY5LTLSFU3D5DIZH2NZLYAKUEHZ4UHDA, seg_pos={0 0}, upload=476992]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu): protocol: expected piece hash; context canceled; EOF Dec 28 07:24:34.566169999 (543h55m34.437253044s) [noise=true , address=77.90.31.13:4626 , piece_id=7DPB5UBH7ABVQEHEZAJRVY5LTLSFU3D5DIZH2NZLYAKUEHZ4UHDA, seg_pos={0 0}, upload=476992]: Long tail cancellation due to optimal threshold Dec 28 07:24:34.566514885 (543h55m34.437597921s) [noise=true , address=45.91.116.5:4297 , piece_id=TRXYL52PWBQI3U3H5S7XFNEVTIKZXJXVUL7B5H3YL7RWQNVAWSDA, seg_pos={0 0}, upload=476992]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: write tcp 188.166.225.170:41944->45.91.116.5:4297: use of closed network connection; write tcp 188.166.225.170:41944->45.91.116.5:4297: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41944->45.91.116.5:4297: use of closed network connection Dec 28 07:24:34.566522660 (543h55m34.437605695s) [noise=true , address=45.91.116.5:4297 , piece_id=TRXYL52PWBQI3U3H5S7XFNEVTIKZXJXVUL7B5H3YL7RWQNVAWSDA, seg_pos={0 0}, upload=476992]: Long tail cancellation due to optimal threshold Dec 28 07:24:34.566883597 (543h55m34.437966632s) [noise=true , address=45.91.116.8:4046 , piece_id=2FECWK73M4LDD3WHQTQY53KETVJY4FHMSDABWBLOLODMZIJZVNFQ, seg_pos={0 0}, upload=476992]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H): protocol: context canceled; context canceled Dec 28 07:24:34.566890451 (543h55m34.437973486s) [noise=true , address=45.91.116.8:4046 , piece_id=2FECWK73M4LDD3WHQTQY53KETVJY4FHMSDABWBLOLODMZIJZVNFQ, seg_pos={0 0}, upload=476992]: Long tail cancellation due to optimal threshold Dec 28 07:24:34.566907002 (543h55m34.437990037s) [seg_pos={0 0}, upload=476992]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 07:24:34.567001582 (543h55m34.438084617s) [upload=476992]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 07:24:34.567006662 (543h55m34.438089697s) [upload=476992]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 07:24:34.862454529 (543h55m34.733537564s) [upload=476992]: Done uploading segments... Dec 28 07:24:34.862480950 (543h55m34.733563985s) [upload=476992]: Upload finished. err: <nil> Duration: 5.764481546s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 27 out of 60 is failed (threshold=6) [X...XX......X.X..X..XX.XXXXXXX.X...X..X.XX.....X.X.XX.X.X.X.] duration: 5.001251712s trace: https://jaeger.dev.storj.io/trace/6bee2b84b91d624b FAIL (2025-12-28 07:30:38 UTC) orbiter warning: context deadline exceeded Duration: 5.001278243s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 27 out of 60 is failed (threshold=6) [.X...XX......X.X..X..XX.XXXXXXX.X...X..X.XX.....X.X.XX.X.X.X] duration: 5.001404323s trace: https://jaeger.dev.storj.io/trace/6bee2b84b91d624b FAIL (2025-12-28 07:30:20 UTC) orbiter warning: context deadline exceeded Duration: 5.001425442s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [.....X...XXX.XX.............XX.X...X...X.XXX..X.XX.X........] duration: 6.195942434s trace: https://jaeger.dev.storj.io/trace/19034703e3293256 FAIL (2025-12-28 07:30:18 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 28 07:24:31.869041095 (543h55m31.740124131s) [upload=476992]: Starting upload Dec 28 07:24:31.869091793 (543h55m31.740174838s) [upload=476992]: Done starting upload Dec 28 07:24:31.869203846 (543h55m31.740286901s) [upload=476992]: Uploading segments... Dec 28 07:24:31.869341798 (543h55m31.740424854s) [upload=476992]: Got next segment. Inline: false Dec 28 07:24:31.869387505 (543h55m31.74047054s) [upload=476992]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 07:24:31.869392474 (543h55m31.74047551s) [upload=476992]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 07:24:32.112124481 (543h55m31.983207526s) [seg_pos={0 0}, upload=476992]: Begin upload segment... Dec 28 07:24:32.112187041 (543h55m31.983270086s) [seg_pos={0 0}, upload=476992]: Done begin upload segment. Dec 28 07:24:32.112262514 (543h55m31.983345549s) [noise=true , address=145.79.227.45:4231 , piece_id=M4CCTNOTHYYE2YXIGKONI26OH7W2I4R3IQJH2TXY6ZFKLRYTBPNA, seg_pos={0 0}, upload=476992]: Uploading piece... Dec 28 07:24:32.112347206 (543h55m31.983430251s) [noise=true , address=145.79.227.31:4798 , piece_id=6MA7SUPGRYZ7RZSIURS3IFEE4B7VRDFWLYVTWCQFHSYMGX5ULP6A, seg_pos={0 0}, upload=476992]: Uploading piece... Dec 28 07:24:32.112868147 (543h55m31.983951192s) [noise=true , address=145.79.227.7:4293 , piece_id=ILHS7U65IXGQUHGUZOSKDLR75D2ADT6VMY2DF3Q44JGA6NC35YQA, seg_pos={0 0}, upload=476992]: Uploading piece... Dec 28 07:24:32.113275393 (543h55m31.984358438s) [noise=true , address=145.79.227.17:4972 , piece_id=2DEF4NOZF5IEFEHLS3C245RYLN6TKKHIEYAVXJWK2EZDTDB5LZ6A, seg_pos={0 0}, upload=476992]: Uploading piece... Dec 28 07:24:32.114689235 (543h55m31.98577228s) [noise=true , address=45.91.116.5:4297 , piece_id=TRXYL52PWBQI3U3H5S7XFNEVTIKZXJXVUL7B5H3YL7RWQNVAWSDA, seg_pos={0 0}, upload=476992]: Uploading piece... Dec 28 07:24:32.115243370 (543h55m31.986326416s) [noise=true , address=145.79.227.35:4481 , piece_id=45CJ7I6U6OJLKJNNCECWBTRWKL3MDXPAP5TSWR57JIXAQ7QTF5CQ, seg_pos={0 0}, upload=476992]: Uploading piece... Dec 28 07:24:32.115709027 (543h55m31.986792072s) [noise=true , address=45.91.116.8:4046 , piece_id=2FECWK73M4LDD3WHQTQY53KETVJY4FHMSDABWBLOLODMZIJZVNFQ, seg_pos={0 0}, upload=476992]: Uploading piece... Dec 28 07:24:32.116664757 (543h55m31.987747792s) [noise=true , address=45.91.116.12:4225 , piece_id=FEEGNTEUKADXJA6Q525AUFVFNTDAWYGP5HFEVNVJN6Q4CQN2BB3A, seg_pos={0 0}, upload=476992]: Uploading piece... Dec 28 07:24:32.117199886 (543h55m31.988282931s) [noise=true , address=145.79.227.14:4839 , piece_id=HWBXXJKYTV7C6TOBWMK7U5QJVRROMVPLZLPAP7URZ2E6J6TKBOVA, seg_pos={0 0}, upload=476992]: Uploading piece... Dec 28 07:24:32.118394961 (543h55m31.989477996s) [noise=true , address=145.79.227.37:4567 , piece_id=OPWBMOJSFIVAV2ZQVTSPIUU3RWB6R35COP75KWQW7N4EM3EUE3MA, seg_pos={0 0}, upload=476992]: Uploading piece... Dec 28 07:24:32.119110775 (543h55m31.99019382s) [noise=true , address=77.90.31.13:4626 , piece_id=7DPB5UBH7ABVQEHEZAJRVY5LTLSFU3D5DIZH2NZLYAKUEHZ4UHDA, seg_pos={0 0}, upload=476992]: Uploading piece... Dec 28 07:24:32.122486352 (543h55m31.993569397s) [noise=true , address=45.91.116.13:4156 , piece_id=R7RVEBAT5VGQZHUHRLEPBS7STP7XVA6AY4YSLPRIXJKUVKCFF4VQ, seg_pos={0 0}, upload=476992]: Uploading piece... Dec 28 07:24:32.124638781 (543h55m31.995721816s) [noise=true , address=145.79.227.16:4911 , piece_id=HDA2REQUIC2Z7XRKP77UDNOWFVITNSXSMNFQA7NHIVYJNBB6SIZQ, seg_pos={0 0}, upload=476992]: Uploading piece... Dec 28 07:24:32.124849412 (543h55m31.995932457s) [noise=true , address=145.79.227.18:4007 , piece_id=TZ6KLDXOBKVHULQB3E4WLXRACWPTA6C5JZFWITYFFSXK2D5KOCXA, seg_pos={0 0}, upload=476992]: Uploading piece... Dec 28 07:24:32.126406407 (543h55m31.997489442s) [noise=true , address=45.91.116.10:4084 , piece_id=HVGRMOQXOHAFUNUXY2J42H5OJL6XDEMLO3BLROQEFJ6INUAK4I4A, seg_pos={0 0}, upload=476992]: Uploading piece... Dec 28 07:24:32.127679831 (543h55m31.998762886s) [noise=true , address=145.79.227.42:4123 , piece_id=TIHM4TDHHEXOITN4UL6ZTQUGPD3KU67QNH5Y2HZX7IYQ6EBHKXUQ, seg_pos={0 0}, upload=476992]: Uploading piece... Dec 28 07:24:33.749682045 (543h55m33.62076508s) [upload=476992]: Next returned nil segment Dec 28 07:24:33.749699568 (543h55m33.620782613s) [upload=476992]: Waiting for error group managing segments... Dec 28 07:24:34.119458798 (543h55m33.990541843s) [noise=true , address=145.79.227.45:4231 , piece_id=M4CCTNOTHYYE2YXIGKONI26OH7W2I4R3IQJH2TXY6ZFKLRYTBPNA, seg_pos={0 0}, upload=476992]: Done uploading piece. err: <nil> Dec 28 07:24:34.121601728 (543h55m33.992684763s) [noise=true , address=145.79.227.17:4972 , piece_id=2DEF4NOZF5IEFEHLS3C245RYLN6TKKHIEYAVXJWK2EZDTDB5LZ6A, seg_pos={0 0}, upload=476992]: Done uploading piece. err: <nil> Dec 28 07:24:34.128325031 (543h55m33.999408066s) [noise=true , address=145.79.227.37:4567 , piece_id=OPWBMOJSFIVAV2ZQVTSPIUU3RWB6R35COP75KWQW7N4EM3EUE3MA, seg_pos={0 0}, upload=476992]: Done uploading piece. err: <nil> Dec 28 07:24:34.128912158 (543h55m33.999995203s) [noise=true , address=145.79.227.16:4911 , piece_id=HDA2REQUIC2Z7XRKP77UDNOWFVITNSXSMNFQA7NHIVYJNBB6SIZQ, seg_pos={0 0}, upload=476992]: Done uploading piece. err: <nil> Dec 28 07:24:34.131137215 (543h55m34.00222025s) [noise=true , address=145.79.227.42:4123 , piece_id=TIHM4TDHHEXOITN4UL6ZTQUGPD3KU67QNH5Y2HZX7IYQ6EBHKXUQ, seg_pos={0 0}, upload=476992]: Done uploading piece. err: <nil> Dec 28 07:24:34.133603162 (543h55m34.004686197s) [noise=true , address=145.79.227.7:4293 , piece_id=ILHS7U65IXGQUHGUZOSKDLR75D2ADT6VMY2DF3Q44JGA6NC35YQA, seg_pos={0 0}, upload=476992]: Done uploading piece. err: <nil> Dec 28 07:24:34.140904865 (543h55m34.0119879s) [noise=true , address=145.79.227.35:4481 , piece_id=45CJ7I6U6OJLKJNNCECWBTRWKL3MDXPAP5TSWR57JIXAQ7QTF5CQ, seg_pos={0 0}, upload=476992]: Done uploading piece. err: <nil> Dec 28 07:24:34.146817494 (543h55m34.017900539s) [noise=true , address=145.79.227.14:4839 , piece_id=HWBXXJKYTV7C6TOBWMK7U5QJVRROMVPLZLPAP7URZ2E6J6TKBOVA, seg_pos={0 0}, upload=476992]: Done uploading piece. err: <nil> Dec 28 07:24:34.146901404 (543h55m34.017984449s) [noise=true , address=145.79.227.31:4798 , piece_id=6MA7SUPGRYZ7RZSIURS3IFEE4B7VRDFWLYVTWCQFHSYMGX5ULP6A, seg_pos={0 0}, upload=476992]: Done uploading piece. err: <nil> Dec 28 07:24:34.154751622 (543h55m34.025834657s) [noise=true , address=145.79.227.18:4007 , piece_id=TZ6KLDXOBKVHULQB3E4WLXRACWPTA6C5JZFWITYFFSXK2D5KOCXA, seg_pos={0 0}, upload=476992]: Done uploading piece. err: <nil> Dec 28 07:24:34.362215111 (543h55m34.233298146s) [noise=true , address=45.91.116.12:4225 , piece_id=FEEGNTEUKADXJA6Q525AUFVFNTDAWYGP5HFEVNVJN6Q4CQN2BB3A, seg_pos={0 0}, upload=476992]: Done uploading piece. err: <nil> Dec 28 07:24:34.372017326 (543h55m34.243100371s) [noise=true , address=45.91.116.13:4156 , piece_id=R7RVEBAT5VGQZHUHRLEPBS7STP7XVA6AY4YSLPRIXJKUVKCFF4VQ, seg_pos={0 0}, upload=476992]: Done uploading piece. err: <nil> Dec 28 07:24:34.565728377 (543h55m34.436811422s) [noise=true , address=45.91.116.10:4084 , piece_id=HVGRMOQXOHAFUNUXY2J42H5OJL6XDEMLO3BLROQEFJ6INUAK4I4A, seg_pos={0 0}, upload=476992]: Done uploading piece. err: <nil> Dec 28 07:24:34.565747725 (543h55m34.43683076s) [seg_pos={0 0}, upload=476992]: Segment reached optimal threshold of 13 pieces. Dec 28 07:24:34.566162044 (543h55m34.437245089s) [noise=true , address=77.90.31.13:4626 , piece_id=7DPB5UBH7ABVQEHEZAJRVY5LTLSFU3D5DIZH2NZLYAKUEHZ4UHDA, seg_pos={0 0}, upload=476992]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu): protocol: expected piece hash; context canceled; EOF Dec 28 07:24:34.566169999 (543h55m34.437253044s) [noise=true , address=77.90.31.13:4626 , piece_id=7DPB5UBH7ABVQEHEZAJRVY5LTLSFU3D5DIZH2NZLYAKUEHZ4UHDA, seg_pos={0 0}, upload=476992]: Long tail cancellation due to optimal threshold Dec 28 07:24:34.566514885 (543h55m34.437597921s) [noise=true , address=45.91.116.5:4297 , piece_id=TRXYL52PWBQI3U3H5S7XFNEVTIKZXJXVUL7B5H3YL7RWQNVAWSDA, seg_pos={0 0}, upload=476992]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: write tcp 188.166.225.170:41944->45.91.116.5:4297: use of closed network connection; write tcp 188.166.225.170:41944->45.91.116.5:4297: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41944->45.91.116.5:4297: use of closed network connection Dec 28 07:24:34.566522660 (543h55m34.437605695s) [noise=true , address=45.91.116.5:4297 , piece_id=TRXYL52PWBQI3U3H5S7XFNEVTIKZXJXVUL7B5H3YL7RWQNVAWSDA, seg_pos={0 0}, upload=476992]: Long tail cancellation due to optimal threshold Dec 28 07:24:34.566883597 (543h55m34.437966632s) [noise=true , address=45.91.116.8:4046 , piece_id=2FECWK73M4LDD3WHQTQY53KETVJY4FHMSDABWBLOLODMZIJZVNFQ, seg_pos={0 0}, upload=476992]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H): protocol: context canceled; context canceled Dec 28 07:24:34.566890451 (543h55m34.437973486s) [noise=true , address=45.91.116.8:4046 , piece_id=2FECWK73M4LDD3WHQTQY53KETVJY4FHMSDABWBLOLODMZIJZVNFQ, seg_pos={0 0}, upload=476992]: Long tail cancellation due to optimal threshold Dec 28 07:24:34.566907002 (543h55m34.437990037s) [seg_pos={0 0}, upload=476992]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 07:24:34.567001582 (543h55m34.438084617s) [upload=476992]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 07:24:34.567006662 (543h55m34.438089697s) [upload=476992]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 07:24:34.862454529 (543h55m34.733537564s) [upload=476992]: Done uploading segments... Dec 28 07:24:34.862480950 (543h55m34.733563985s) [upload=476992]: Upload finished. err: <nil> Duration: 6.195974314s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 27 out of 60 is failed (threshold=6) [X.X...XX......X.X..X..XX.XXXXXXX.X...X..X.XX.....X.X.XX.X.X.] duration: 5.000609248s trace: https://jaeger.dev.storj.io/trace/28c67882551d1e68 FAIL (2025-12-28 07:30:01 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000639777s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [X.....X...XXX.XX.............XX.X...X...X.XXX..X.XX.X.......] duration: 6.063904199s trace: https://jaeger.dev.storj.io/trace/19034703e3293256 FAIL (2025-12-28 07:29:49 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 28 07:24:31.869041095 (543h55m31.740124131s) [upload=476992]: Starting upload Dec 28 07:24:31.869091793 (543h55m31.740174838s) [upload=476992]: Done starting upload Dec 28 07:24:31.869203846 (543h55m31.740286901s) [upload=476992]: Uploading segments... Dec 28 07:24:31.869341798 (543h55m31.740424854s) [upload=476992]: Got next segment. Inline: false Dec 28 07:24:31.869387505 (543h55m31.74047054s) [upload=476992]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 07:24:31.869392474 (543h55m31.74047551s) [upload=476992]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 07:24:32.112124481 (543h55m31.983207526s) [seg_pos={0 0}, upload=476992]: Begin upload segment... Dec 28 07:24:32.112187041 (543h55m31.983270086s) [seg_pos={0 0}, upload=476992]: Done begin upload segment. Dec 28 07:24:32.112262514 (543h55m31.983345549s) [noise=true , address=145.79.227.45:4231 , piece_id=M4CCTNOTHYYE2YXIGKONI26OH7W2I4R3IQJH2TXY6ZFKLRYTBPNA, seg_pos={0 0}, upload=476992]: Uploading piece... Dec 28 07:24:32.112347206 (543h55m31.983430251s) [noise=true , address=145.79.227.31:4798 , piece_id=6MA7SUPGRYZ7RZSIURS3IFEE4B7VRDFWLYVTWCQFHSYMGX5ULP6A, seg_pos={0 0}, upload=476992]: Uploading piece... Dec 28 07:24:32.112868147 (543h55m31.983951192s) [noise=true , address=145.79.227.7:4293 , piece_id=ILHS7U65IXGQUHGUZOSKDLR75D2ADT6VMY2DF3Q44JGA6NC35YQA, seg_pos={0 0}, upload=476992]: Uploading piece... Dec 28 07:24:32.113275393 (543h55m31.984358438s) [noise=true , address=145.79.227.17:4972 , piece_id=2DEF4NOZF5IEFEHLS3C245RYLN6TKKHIEYAVXJWK2EZDTDB5LZ6A, seg_pos={0 0}, upload=476992]: Uploading piece... Dec 28 07:24:32.114689235 (543h55m31.98577228s) [noise=true , address=45.91.116.5:4297 , piece_id=TRXYL52PWBQI3U3H5S7XFNEVTIKZXJXVUL7B5H3YL7RWQNVAWSDA, seg_pos={0 0}, upload=476992]: Uploading piece... Dec 28 07:24:32.115243370 (543h55m31.986326416s) [noise=true , address=145.79.227.35:4481 , piece_id=45CJ7I6U6OJLKJNNCECWBTRWKL3MDXPAP5TSWR57JIXAQ7QTF5CQ, seg_pos={0 0}, upload=476992]: Uploading piece... Dec 28 07:24:32.115709027 (543h55m31.986792072s) [noise=true , address=45.91.116.8:4046 , piece_id=2FECWK73M4LDD3WHQTQY53KETVJY4FHMSDABWBLOLODMZIJZVNFQ, seg_pos={0 0}, upload=476992]: Uploading piece... Dec 28 07:24:32.116664757 (543h55m31.987747792s) [noise=true , address=45.91.116.12:4225 , piece_id=FEEGNTEUKADXJA6Q525AUFVFNTDAWYGP5HFEVNVJN6Q4CQN2BB3A, seg_pos={0 0}, upload=476992]: Uploading piece... Dec 28 07:24:32.117199886 (543h55m31.988282931s) [noise=true , address=145.79.227.14:4839 , piece_id=HWBXXJKYTV7C6TOBWMK7U5QJVRROMVPLZLPAP7URZ2E6J6TKBOVA, seg_pos={0 0}, upload=476992]: Uploading piece... Dec 28 07:24:32.118394961 (543h55m31.989477996s) [noise=true , address=145.79.227.37:4567 , piece_id=OPWBMOJSFIVAV2ZQVTSPIUU3RWB6R35COP75KWQW7N4EM3EUE3MA, seg_pos={0 0}, upload=476992]: Uploading piece... Dec 28 07:24:32.119110775 (543h55m31.99019382s) [noise=true , address=77.90.31.13:4626 , piece_id=7DPB5UBH7ABVQEHEZAJRVY5LTLSFU3D5DIZH2NZLYAKUEHZ4UHDA, seg_pos={0 0}, upload=476992]: Uploading piece... Dec 28 07:24:32.122486352 (543h55m31.993569397s) [noise=true , address=45.91.116.13:4156 , piece_id=R7RVEBAT5VGQZHUHRLEPBS7STP7XVA6AY4YSLPRIXJKUVKCFF4VQ, seg_pos={0 0}, upload=476992]: Uploading piece... Dec 28 07:24:32.124638781 (543h55m31.995721816s) [noise=true , address=145.79.227.16:4911 , piece_id=HDA2REQUIC2Z7XRKP77UDNOWFVITNSXSMNFQA7NHIVYJNBB6SIZQ, seg_pos={0 0}, upload=476992]: Uploading piece... Dec 28 07:24:32.124849412 (543h55m31.995932457s) [noise=true , address=145.79.227.18:4007 , piece_id=TZ6KLDXOBKVHULQB3E4WLXRACWPTA6C5JZFWITYFFSXK2D5KOCXA, seg_pos={0 0}, upload=476992]: Uploading piece... Dec 28 07:24:32.126406407 (543h55m31.997489442s) [noise=true , address=45.91.116.10:4084 , piece_id=HVGRMOQXOHAFUNUXY2J42H5OJL6XDEMLO3BLROQEFJ6INUAK4I4A, seg_pos={0 0}, upload=476992]: Uploading piece... Dec 28 07:24:32.127679831 (543h55m31.998762886s) [noise=true , address=145.79.227.42:4123 , piece_id=TIHM4TDHHEXOITN4UL6ZTQUGPD3KU67QNH5Y2HZX7IYQ6EBHKXUQ, seg_pos={0 0}, upload=476992]: Uploading piece... Dec 28 07:24:33.749682045 (543h55m33.62076508s) [upload=476992]: Next returned nil segment Dec 28 07:24:33.749699568 (543h55m33.620782613s) [upload=476992]: Waiting for error group managing segments... Dec 28 07:24:34.119458798 (543h55m33.990541843s) [noise=true , address=145.79.227.45:4231 , piece_id=M4CCTNOTHYYE2YXIGKONI26OH7W2I4R3IQJH2TXY6ZFKLRYTBPNA, seg_pos={0 0}, upload=476992]: Done uploading piece. err: <nil> Dec 28 07:24:34.121601728 (543h55m33.992684763s) [noise=true , address=145.79.227.17:4972 , piece_id=2DEF4NOZF5IEFEHLS3C245RYLN6TKKHIEYAVXJWK2EZDTDB5LZ6A, seg_pos={0 0}, upload=476992]: Done uploading piece. err: <nil> Dec 28 07:24:34.128325031 (543h55m33.999408066s) [noise=true , address=145.79.227.37:4567 , piece_id=OPWBMOJSFIVAV2ZQVTSPIUU3RWB6R35COP75KWQW7N4EM3EUE3MA, seg_pos={0 0}, upload=476992]: Done uploading piece. err: <nil> Dec 28 07:24:34.128912158 (543h55m33.999995203s) [noise=true , address=145.79.227.16:4911 , piece_id=HDA2REQUIC2Z7XRKP77UDNOWFVITNSXSMNFQA7NHIVYJNBB6SIZQ, seg_pos={0 0}, upload=476992]: Done uploading piece. err: <nil> Dec 28 07:24:34.131137215 (543h55m34.00222025s) [noise=true , address=145.79.227.42:4123 , piece_id=TIHM4TDHHEXOITN4UL6ZTQUGPD3KU67QNH5Y2HZX7IYQ6EBHKXUQ, seg_pos={0 0}, upload=476992]: Done uploading piece. err: <nil> Dec 28 07:24:34.133603162 (543h55m34.004686197s) [noise=true , address=145.79.227.7:4293 , piece_id=ILHS7U65IXGQUHGUZOSKDLR75D2ADT6VMY2DF3Q44JGA6NC35YQA, seg_pos={0 0}, upload=476992]: Done uploading piece. err: <nil> Dec 28 07:24:34.140904865 (543h55m34.0119879s) [noise=true , address=145.79.227.35:4481 , piece_id=45CJ7I6U6OJLKJNNCECWBTRWKL3MDXPAP5TSWR57JIXAQ7QTF5CQ, seg_pos={0 0}, upload=476992]: Done uploading piece. err: <nil> Dec 28 07:24:34.146817494 (543h55m34.017900539s) [noise=true , address=145.79.227.14:4839 , piece_id=HWBXXJKYTV7C6TOBWMK7U5QJVRROMVPLZLPAP7URZ2E6J6TKBOVA, seg_pos={0 0}, upload=476992]: Done uploading piece. err: <nil> Dec 28 07:24:34.146901404 (543h55m34.017984449s) [noise=true , address=145.79.227.31:4798 , piece_id=6MA7SUPGRYZ7RZSIURS3IFEE4B7VRDFWLYVTWCQFHSYMGX5ULP6A, seg_pos={0 0}, upload=476992]: Done uploading piece. err: <nil> Dec 28 07:24:34.154751622 (543h55m34.025834657s) [noise=true , address=145.79.227.18:4007 , piece_id=TZ6KLDXOBKVHULQB3E4WLXRACWPTA6C5JZFWITYFFSXK2D5KOCXA, seg_pos={0 0}, upload=476992]: Done uploading piece. err: <nil> Dec 28 07:24:34.362215111 (543h55m34.233298146s) [noise=true , address=45.91.116.12:4225 , piece_id=FEEGNTEUKADXJA6Q525AUFVFNTDAWYGP5HFEVNVJN6Q4CQN2BB3A, seg_pos={0 0}, upload=476992]: Done uploading piece. err: <nil> Dec 28 07:24:34.372017326 (543h55m34.243100371s) [noise=true , address=45.91.116.13:4156 , piece_id=R7RVEBAT5VGQZHUHRLEPBS7STP7XVA6AY4YSLPRIXJKUVKCFF4VQ, seg_pos={0 0}, upload=476992]: Done uploading piece. err: <nil> Dec 28 07:24:34.565728377 (543h55m34.436811422s) [noise=true , address=45.91.116.10:4084 , piece_id=HVGRMOQXOHAFUNUXY2J42H5OJL6XDEMLO3BLROQEFJ6INUAK4I4A, seg_pos={0 0}, upload=476992]: Done uploading piece. err: <nil> Dec 28 07:24:34.565747725 (543h55m34.43683076s) [seg_pos={0 0}, upload=476992]: Segment reached optimal threshold of 13 pieces. Dec 28 07:24:34.566162044 (543h55m34.437245089s) [noise=true , address=77.90.31.13:4626 , piece_id=7DPB5UBH7ABVQEHEZAJRVY5LTLSFU3D5DIZH2NZLYAKUEHZ4UHDA, seg_pos={0 0}, upload=476992]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu): protocol: expected piece hash; context canceled; EOF Dec 28 07:24:34.566169999 (543h55m34.437253044s) [noise=true , address=77.90.31.13:4626 , piece_id=7DPB5UBH7ABVQEHEZAJRVY5LTLSFU3D5DIZH2NZLYAKUEHZ4UHDA, seg_pos={0 0}, upload=476992]: Long tail cancellation due to optimal threshold Dec 28 07:24:34.566514885 (543h55m34.437597921s) [noise=true , address=45.91.116.5:4297 , piece_id=TRXYL52PWBQI3U3H5S7XFNEVTIKZXJXVUL7B5H3YL7RWQNVAWSDA, seg_pos={0 0}, upload=476992]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: write tcp 188.166.225.170:41944->45.91.116.5:4297: use of closed network connection; write tcp 188.166.225.170:41944->45.91.116.5:4297: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41944->45.91.116.5:4297: use of closed network connection Dec 28 07:24:34.566522660 (543h55m34.437605695s) [noise=true , address=45.91.116.5:4297 , piece_id=TRXYL52PWBQI3U3H5S7XFNEVTIKZXJXVUL7B5H3YL7RWQNVAWSDA, seg_pos={0 0}, upload=476992]: Long tail cancellation due to optimal threshold Dec 28 07:24:34.566883597 (543h55m34.437966632s) [noise=true , address=45.91.116.8:4046 , piece_id=2FECWK73M4LDD3WHQTQY53KETVJY4FHMSDABWBLOLODMZIJZVNFQ, seg_pos={0 0}, upload=476992]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H): protocol: context canceled; context canceled Dec 28 07:24:34.566890451 (543h55m34.437973486s) [noise=true , address=45.91.116.8:4046 , piece_id=2FECWK73M4LDD3WHQTQY53KETVJY4FHMSDABWBLOLODMZIJZVNFQ, seg_pos={0 0}, upload=476992]: Long tail cancellation due to optimal threshold Dec 28 07:24:34.566907002 (543h55m34.437990037s) [seg_pos={0 0}, upload=476992]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 07:24:34.567001582 (543h55m34.438084617s) [upload=476992]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 07:24:34.567006662 (543h55m34.438089697s) [upload=476992]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 07:24:34.862454529 (543h55m34.733537564s) [upload=476992]: Done uploading segments... Dec 28 07:24:34.862480950 (543h55m34.733563985s) [upload=476992]: Upload finished. err: <nil> Duration: 6.063932383s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 27 out of 60 is failed (threshold=6) [.X.X...XX......X.X..X..XX.XXXXXXX.X...X..X.XX.....X.X.XX.X.X] duration: 5.001408769s trace: https://jaeger.dev.storj.io/trace/28c67882551d1e68 FAIL (2025-12-28 07:29:27 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001433236s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 26 out of 60 is failed (threshold=6) [..X.X...XX......X.X..X..XX.XXXXXXX.X...X..X.XX.....X.X.XX.X.] duration: 4.263300915s trace: https://jaeger.dev.storj.io/trace/35783acb2682fb57 FAIL (2025-12-28 07:29:08 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.263348776s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [XX.....X...XXX.XX.............XX.X...X...X.XXX..X.XX.X......] duration: 5.693617549s trace: https://jaeger.dev.storj.io/trace/19034703e3293256 FAIL (2025-12-28 07:28:38 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 28 07:24:31.869041095 (543h55m31.740124131s) [upload=476992]: Starting upload Dec 28 07:24:31.869091793 (543h55m31.740174838s) [upload=476992]: Done starting upload Dec 28 07:24:31.869203846 (543h55m31.740286901s) [upload=476992]: Uploading segments... Dec 28 07:24:31.869341798 (543h55m31.740424854s) [upload=476992]: Got next segment. Inline: false Dec 28 07:24:31.869387505 (543h55m31.74047054s) [upload=476992]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 07:24:31.869392474 (543h55m31.74047551s) [upload=476992]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 07:24:32.112124481 (543h55m31.983207526s) [seg_pos={0 0}, upload=476992]: Begin upload segment... Dec 28 07:24:32.112187041 (543h55m31.983270086s) [seg_pos={0 0}, upload=476992]: Done begin upload segment. Dec 28 07:24:32.112262514 (543h55m31.983345549s) [noise=true , address=145.79.227.45:4231 , piece_id=M4CCTNOTHYYE2YXIGKONI26OH7W2I4R3IQJH2TXY6ZFKLRYTBPNA, seg_pos={0 0}, upload=476992]: Uploading piece... Dec 28 07:24:32.112347206 (543h55m31.983430251s) [noise=true , address=145.79.227.31:4798 , piece_id=6MA7SUPGRYZ7RZSIURS3IFEE4B7VRDFWLYVTWCQFHSYMGX5ULP6A, seg_pos={0 0}, upload=476992]: Uploading piece... Dec 28 07:24:32.112868147 (543h55m31.983951192s) [noise=true , address=145.79.227.7:4293 , piece_id=ILHS7U65IXGQUHGUZOSKDLR75D2ADT6VMY2DF3Q44JGA6NC35YQA, seg_pos={0 0}, upload=476992]: Uploading piece... Dec 28 07:24:32.113275393 (543h55m31.984358438s) [noise=true , address=145.79.227.17:4972 , piece_id=2DEF4NOZF5IEFEHLS3C245RYLN6TKKHIEYAVXJWK2EZDTDB5LZ6A, seg_pos={0 0}, upload=476992]: Uploading piece... Dec 28 07:24:32.114689235 (543h55m31.98577228s) [noise=true , address=45.91.116.5:4297 , piece_id=TRXYL52PWBQI3U3H5S7XFNEVTIKZXJXVUL7B5H3YL7RWQNVAWSDA, seg_pos={0 0}, upload=476992]: Uploading piece... Dec 28 07:24:32.115243370 (543h55m31.986326416s) [noise=true , address=145.79.227.35:4481 , piece_id=45CJ7I6U6OJLKJNNCECWBTRWKL3MDXPAP5TSWR57JIXAQ7QTF5CQ, seg_pos={0 0}, upload=476992]: Uploading piece... Dec 28 07:24:32.115709027 (543h55m31.986792072s) [noise=true , address=45.91.116.8:4046 , piece_id=2FECWK73M4LDD3WHQTQY53KETVJY4FHMSDABWBLOLODMZIJZVNFQ, seg_pos={0 0}, upload=476992]: Uploading piece... Dec 28 07:24:32.116664757 (543h55m31.987747792s) [noise=true , address=45.91.116.12:4225 , piece_id=FEEGNTEUKADXJA6Q525AUFVFNTDAWYGP5HFEVNVJN6Q4CQN2BB3A, seg_pos={0 0}, upload=476992]: Uploading piece... Dec 28 07:24:32.117199886 (543h55m31.988282931s) [noise=true , address=145.79.227.14:4839 , piece_id=HWBXXJKYTV7C6TOBWMK7U5QJVRROMVPLZLPAP7URZ2E6J6TKBOVA, seg_pos={0 0}, upload=476992]: Uploading piece... Dec 28 07:24:32.118394961 (543h55m31.989477996s) [noise=true , address=145.79.227.37:4567 , piece_id=OPWBMOJSFIVAV2ZQVTSPIUU3RWB6R35COP75KWQW7N4EM3EUE3MA, seg_pos={0 0}, upload=476992]: Uploading piece... Dec 28 07:24:32.119110775 (543h55m31.99019382s) [noise=true , address=77.90.31.13:4626 , piece_id=7DPB5UBH7ABVQEHEZAJRVY5LTLSFU3D5DIZH2NZLYAKUEHZ4UHDA, seg_pos={0 0}, upload=476992]: Uploading piece... Dec 28 07:24:32.122486352 (543h55m31.993569397s) [noise=true , address=45.91.116.13:4156 , piece_id=R7RVEBAT5VGQZHUHRLEPBS7STP7XVA6AY4YSLPRIXJKUVKCFF4VQ, seg_pos={0 0}, upload=476992]: Uploading piece... Dec 28 07:24:32.124638781 (543h55m31.995721816s) [noise=true , address=145.79.227.16:4911 , piece_id=HDA2REQUIC2Z7XRKP77UDNOWFVITNSXSMNFQA7NHIVYJNBB6SIZQ, seg_pos={0 0}, upload=476992]: Uploading piece... Dec 28 07:24:32.124849412 (543h55m31.995932457s) [noise=true , address=145.79.227.18:4007 , piece_id=TZ6KLDXOBKVHULQB3E4WLXRACWPTA6C5JZFWITYFFSXK2D5KOCXA, seg_pos={0 0}, upload=476992]: Uploading piece... Dec 28 07:24:32.126406407 (543h55m31.997489442s) [noise=true , address=45.91.116.10:4084 , piece_id=HVGRMOQXOHAFUNUXY2J42H5OJL6XDEMLO3BLROQEFJ6INUAK4I4A, seg_pos={0 0}, upload=476992]: Uploading piece... Dec 28 07:24:32.127679831 (543h55m31.998762886s) [noise=true , address=145.79.227.42:4123 , piece_id=TIHM4TDHHEXOITN4UL6ZTQUGPD3KU67QNH5Y2HZX7IYQ6EBHKXUQ, seg_pos={0 0}, upload=476992]: Uploading piece... Dec 28 07:24:33.749682045 (543h55m33.62076508s) [upload=476992]: Next returned nil segment Dec 28 07:24:33.749699568 (543h55m33.620782613s) [upload=476992]: Waiting for error group managing segments... Dec 28 07:24:34.119458798 (543h55m33.990541843s) [noise=true , address=145.79.227.45:4231 , piece_id=M4CCTNOTHYYE2YXIGKONI26OH7W2I4R3IQJH2TXY6ZFKLRYTBPNA, seg_pos={0 0}, upload=476992]: Done uploading piece. err: <nil> Dec 28 07:24:34.121601728 (543h55m33.992684763s) [noise=true , address=145.79.227.17:4972 , piece_id=2DEF4NOZF5IEFEHLS3C245RYLN6TKKHIEYAVXJWK2EZDTDB5LZ6A, seg_pos={0 0}, upload=476992]: Done uploading piece. err: <nil> Dec 28 07:24:34.128325031 (543h55m33.999408066s) [noise=true , address=145.79.227.37:4567 , piece_id=OPWBMOJSFIVAV2ZQVTSPIUU3RWB6R35COP75KWQW7N4EM3EUE3MA, seg_pos={0 0}, upload=476992]: Done uploading piece. err: <nil> Dec 28 07:24:34.128912158 (543h55m33.999995203s) [noise=true , address=145.79.227.16:4911 , piece_id=HDA2REQUIC2Z7XRKP77UDNOWFVITNSXSMNFQA7NHIVYJNBB6SIZQ, seg_pos={0 0}, upload=476992]: Done uploading piece. err: <nil> Dec 28 07:24:34.131137215 (543h55m34.00222025s) [noise=true , address=145.79.227.42:4123 , piece_id=TIHM4TDHHEXOITN4UL6ZTQUGPD3KU67QNH5Y2HZX7IYQ6EBHKXUQ, seg_pos={0 0}, upload=476992]: Done uploading piece. err: <nil> Dec 28 07:24:34.133603162 (543h55m34.004686197s) [noise=true , address=145.79.227.7:4293 , piece_id=ILHS7U65IXGQUHGUZOSKDLR75D2ADT6VMY2DF3Q44JGA6NC35YQA, seg_pos={0 0}, upload=476992]: Done uploading piece. err: <nil> Dec 28 07:24:34.140904865 (543h55m34.0119879s) [noise=true , address=145.79.227.35:4481 , piece_id=45CJ7I6U6OJLKJNNCECWBTRWKL3MDXPAP5TSWR57JIXAQ7QTF5CQ, seg_pos={0 0}, upload=476992]: Done uploading piece. err: <nil> Dec 28 07:24:34.146817494 (543h55m34.017900539s) [noise=true , address=145.79.227.14:4839 , piece_id=HWBXXJKYTV7C6TOBWMK7U5QJVRROMVPLZLPAP7URZ2E6J6TKBOVA, seg_pos={0 0}, upload=476992]: Done uploading piece. err: <nil> Dec 28 07:24:34.146901404 (543h55m34.017984449s) [noise=true , address=145.79.227.31:4798 , piece_id=6MA7SUPGRYZ7RZSIURS3IFEE4B7VRDFWLYVTWCQFHSYMGX5ULP6A, seg_pos={0 0}, upload=476992]: Done uploading piece. err: <nil> Dec 28 07:24:34.154751622 (543h55m34.025834657s) [noise=true , address=145.79.227.18:4007 , piece_id=TZ6KLDXOBKVHULQB3E4WLXRACWPTA6C5JZFWITYFFSXK2D5KOCXA, seg_pos={0 0}, upload=476992]: Done uploading piece. err: <nil> Dec 28 07:24:34.362215111 (543h55m34.233298146s) [noise=true , address=45.91.116.12:4225 , piece_id=FEEGNTEUKADXJA6Q525AUFVFNTDAWYGP5HFEVNVJN6Q4CQN2BB3A, seg_pos={0 0}, upload=476992]: Done uploading piece. err: <nil> Dec 28 07:24:34.372017326 (543h55m34.243100371s) [noise=true , address=45.91.116.13:4156 , piece_id=R7RVEBAT5VGQZHUHRLEPBS7STP7XVA6AY4YSLPRIXJKUVKCFF4VQ, seg_pos={0 0}, upload=476992]: Done uploading piece. err: <nil> Dec 28 07:24:34.565728377 (543h55m34.436811422s) [noise=true , address=45.91.116.10:4084 , piece_id=HVGRMOQXOHAFUNUXY2J42H5OJL6XDEMLO3BLROQEFJ6INUAK4I4A, seg_pos={0 0}, upload=476992]: Done uploading piece. err: <nil> Dec 28 07:24:34.565747725 (543h55m34.43683076s) [seg_pos={0 0}, upload=476992]: Segment reached optimal threshold of 13 pieces. Dec 28 07:24:34.566162044 (543h55m34.437245089s) [noise=true , address=77.90.31.13:4626 , piece_id=7DPB5UBH7ABVQEHEZAJRVY5LTLSFU3D5DIZH2NZLYAKUEHZ4UHDA, seg_pos={0 0}, upload=476992]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu): protocol: expected piece hash; context canceled; EOF Dec 28 07:24:34.566169999 (543h55m34.437253044s) [noise=true , address=77.90.31.13:4626 , piece_id=7DPB5UBH7ABVQEHEZAJRVY5LTLSFU3D5DIZH2NZLYAKUEHZ4UHDA, seg_pos={0 0}, upload=476992]: Long tail cancellation due to optimal threshold Dec 28 07:24:34.566514885 (543h55m34.437597921s) [noise=true , address=45.91.116.5:4297 , piece_id=TRXYL52PWBQI3U3H5S7XFNEVTIKZXJXVUL7B5H3YL7RWQNVAWSDA, seg_pos={0 0}, upload=476992]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: write tcp 188.166.225.170:41944->45.91.116.5:4297: use of closed network connection; write tcp 188.166.225.170:41944->45.91.116.5:4297: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41944->45.91.116.5:4297: use of closed network connection Dec 28 07:24:34.566522660 (543h55m34.437605695s) [noise=true , address=45.91.116.5:4297 , piece_id=TRXYL52PWBQI3U3H5S7XFNEVTIKZXJXVUL7B5H3YL7RWQNVAWSDA, seg_pos={0 0}, upload=476992]: Long tail cancellation due to optimal threshold Dec 28 07:24:34.566883597 (543h55m34.437966632s) [noise=true , address=45.91.116.8:4046 , piece_id=2FECWK73M4LDD3WHQTQY53KETVJY4FHMSDABWBLOLODMZIJZVNFQ, seg_pos={0 0}, upload=476992]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H): protocol: context canceled; context canceled Dec 28 07:24:34.566890451 (543h55m34.437973486s) [noise=true , address=45.91.116.8:4046 , piece_id=2FECWK73M4LDD3WHQTQY53KETVJY4FHMSDABWBLOLODMZIJZVNFQ, seg_pos={0 0}, upload=476992]: Long tail cancellation due to optimal threshold Dec 28 07:24:34.566907002 (543h55m34.437990037s) [seg_pos={0 0}, upload=476992]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 07:24:34.567001582 (543h55m34.438084617s) [upload=476992]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 07:24:34.567006662 (543h55m34.438089697s) [upload=476992]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 07:24:34.862454529 (543h55m34.733537564s) [upload=476992]: Done uploading segments... Dec 28 07:24:34.862480950 (543h55m34.733563985s) [upload=476992]: Upload finished. err: <nil> Duration: 5.693644099s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 26 out of 60 is failed (threshold=6) [...X.X...XX......X.X..X..XX.XXXXXXX.X...X..X.XX.....X.X.XX.X] duration: 5.001987563s trace: https://jaeger.dev.storj.io/trace/35783acb2682fb57 FAIL (2025-12-28 07:28:16 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.002033882s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [.XX.....X...XXX.XX.............XX.X...X...X.XXX..X.XX.X.....] duration: 5.804226768s trace: https://jaeger.dev.storj.io/trace/19034703e3293256 FAIL (2025-12-28 07:27:46 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 28 07:24:31.869041095 (543h55m31.740124131s) [upload=476992]: Starting upload Dec 28 07:24:31.869091793 (543h55m31.740174838s) [upload=476992]: Done starting upload Dec 28 07:24:31.869203846 (543h55m31.740286901s) [upload=476992]: Uploading segments... Dec 28 07:24:31.869341798 (543h55m31.740424854s) [upload=476992]: Got next segment. Inline: false Dec 28 07:24:31.869387505 (543h55m31.74047054s) [upload=476992]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 07:24:31.869392474 (543h55m31.74047551s) [upload=476992]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 07:24:32.112124481 (543h55m31.983207526s) [seg_pos={0 0}, upload=476992]: Begin upload segment... Dec 28 07:24:32.112187041 (543h55m31.983270086s) [seg_pos={0 0}, upload=476992]: Done begin upload segment. Dec 28 07:24:32.112262514 (543h55m31.983345549s) [noise=true , address=145.79.227.45:4231 , piece_id=M4CCTNOTHYYE2YXIGKONI26OH7W2I4R3IQJH2TXY6ZFKLRYTBPNA, seg_pos={0 0}, upload=476992]: Uploading piece... Dec 28 07:24:32.112347206 (543h55m31.983430251s) [noise=true , address=145.79.227.31:4798 , piece_id=6MA7SUPGRYZ7RZSIURS3IFEE4B7VRDFWLYVTWCQFHSYMGX5ULP6A, seg_pos={0 0}, upload=476992]: Uploading piece... Dec 28 07:24:32.112868147 (543h55m31.983951192s) [noise=true , address=145.79.227.7:4293 , piece_id=ILHS7U65IXGQUHGUZOSKDLR75D2ADT6VMY2DF3Q44JGA6NC35YQA, seg_pos={0 0}, upload=476992]: Uploading piece... Dec 28 07:24:32.113275393 (543h55m31.984358438s) [noise=true , address=145.79.227.17:4972 , piece_id=2DEF4NOZF5IEFEHLS3C245RYLN6TKKHIEYAVXJWK2EZDTDB5LZ6A, seg_pos={0 0}, upload=476992]: Uploading piece... Dec 28 07:24:32.114689235 (543h55m31.98577228s) [noise=true , address=45.91.116.5:4297 , piece_id=TRXYL52PWBQI3U3H5S7XFNEVTIKZXJXVUL7B5H3YL7RWQNVAWSDA, seg_pos={0 0}, upload=476992]: Uploading piece... Dec 28 07:24:32.115243370 (543h55m31.986326416s) [noise=true , address=145.79.227.35:4481 , piece_id=45CJ7I6U6OJLKJNNCECWBTRWKL3MDXPAP5TSWR57JIXAQ7QTF5CQ, seg_pos={0 0}, upload=476992]: Uploading piece... Dec 28 07:24:32.115709027 (543h55m31.986792072s) [noise=true , address=45.91.116.8:4046 , piece_id=2FECWK73M4LDD3WHQTQY53KETVJY4FHMSDABWBLOLODMZIJZVNFQ, seg_pos={0 0}, upload=476992]: Uploading piece... Dec 28 07:24:32.116664757 (543h55m31.987747792s) [noise=true , address=45.91.116.12:4225 , piece_id=FEEGNTEUKADXJA6Q525AUFVFNTDAWYGP5HFEVNVJN6Q4CQN2BB3A, seg_pos={0 0}, upload=476992]: Uploading piece... Dec 28 07:24:32.117199886 (543h55m31.988282931s) [noise=true , address=145.79.227.14:4839 , piece_id=HWBXXJKYTV7C6TOBWMK7U5QJVRROMVPLZLPAP7URZ2E6J6TKBOVA, seg_pos={0 0}, upload=476992]: Uploading piece... Dec 28 07:24:32.118394961 (543h55m31.989477996s) [noise=true , address=145.79.227.37:4567 , piece_id=OPWBMOJSFIVAV2ZQVTSPIUU3RWB6R35COP75KWQW7N4EM3EUE3MA, seg_pos={0 0}, upload=476992]: Uploading piece... Dec 28 07:24:32.119110775 (543h55m31.99019382s) [noise=true , address=77.90.31.13:4626 , piece_id=7DPB5UBH7ABVQEHEZAJRVY5LTLSFU3D5DIZH2NZLYAKUEHZ4UHDA, seg_pos={0 0}, upload=476992]: Uploading piece... Dec 28 07:24:32.122486352 (543h55m31.993569397s) [noise=true , address=45.91.116.13:4156 , piece_id=R7RVEBAT5VGQZHUHRLEPBS7STP7XVA6AY4YSLPRIXJKUVKCFF4VQ, seg_pos={0 0}, upload=476992]: Uploading piece... Dec 28 07:24:32.124638781 (543h55m31.995721816s) [noise=true , address=145.79.227.16:4911 , piece_id=HDA2REQUIC2Z7XRKP77UDNOWFVITNSXSMNFQA7NHIVYJNBB6SIZQ, seg_pos={0 0}, upload=476992]: Uploading piece... Dec 28 07:24:32.124849412 (543h55m31.995932457s) [noise=true , address=145.79.227.18:4007 , piece_id=TZ6KLDXOBKVHULQB3E4WLXRACWPTA6C5JZFWITYFFSXK2D5KOCXA, seg_pos={0 0}, upload=476992]: Uploading piece... Dec 28 07:24:32.126406407 (543h55m31.997489442s) [noise=true , address=45.91.116.10:4084 , piece_id=HVGRMOQXOHAFUNUXY2J42H5OJL6XDEMLO3BLROQEFJ6INUAK4I4A, seg_pos={0 0}, upload=476992]: Uploading piece... Dec 28 07:24:32.127679831 (543h55m31.998762886s) [noise=true , address=145.79.227.42:4123 , piece_id=TIHM4TDHHEXOITN4UL6ZTQUGPD3KU67QNH5Y2HZX7IYQ6EBHKXUQ, seg_pos={0 0}, upload=476992]: Uploading piece... Dec 28 07:24:33.749682045 (543h55m33.62076508s) [upload=476992]: Next returned nil segment Dec 28 07:24:33.749699568 (543h55m33.620782613s) [upload=476992]: Waiting for error group managing segments... Dec 28 07:24:34.119458798 (543h55m33.990541843s) [noise=true , address=145.79.227.45:4231 , piece_id=M4CCTNOTHYYE2YXIGKONI26OH7W2I4R3IQJH2TXY6ZFKLRYTBPNA, seg_pos={0 0}, upload=476992]: Done uploading piece. err: <nil> Dec 28 07:24:34.121601728 (543h55m33.992684763s) [noise=true , address=145.79.227.17:4972 , piece_id=2DEF4NOZF5IEFEHLS3C245RYLN6TKKHIEYAVXJWK2EZDTDB5LZ6A, seg_pos={0 0}, upload=476992]: Done uploading piece. err: <nil> Dec 28 07:24:34.128325031 (543h55m33.999408066s) [noise=true , address=145.79.227.37:4567 , piece_id=OPWBMOJSFIVAV2ZQVTSPIUU3RWB6R35COP75KWQW7N4EM3EUE3MA, seg_pos={0 0}, upload=476992]: Done uploading piece. err: <nil> Dec 28 07:24:34.128912158 (543h55m33.999995203s) [noise=true , address=145.79.227.16:4911 , piece_id=HDA2REQUIC2Z7XRKP77UDNOWFVITNSXSMNFQA7NHIVYJNBB6SIZQ, seg_pos={0 0}, upload=476992]: Done uploading piece. err: <nil> Dec 28 07:24:34.131137215 (543h55m34.00222025s) [noise=true , address=145.79.227.42:4123 , piece_id=TIHM4TDHHEXOITN4UL6ZTQUGPD3KU67QNH5Y2HZX7IYQ6EBHKXUQ, seg_pos={0 0}, upload=476992]: Done uploading piece. err: <nil> Dec 28 07:24:34.133603162 (543h55m34.004686197s) [noise=true , address=145.79.227.7:4293 , piece_id=ILHS7U65IXGQUHGUZOSKDLR75D2ADT6VMY2DF3Q44JGA6NC35YQA, seg_pos={0 0}, upload=476992]: Done uploading piece. err: <nil> Dec 28 07:24:34.140904865 (543h55m34.0119879s) [noise=true , address=145.79.227.35:4481 , piece_id=45CJ7I6U6OJLKJNNCECWBTRWKL3MDXPAP5TSWR57JIXAQ7QTF5CQ, seg_pos={0 0}, upload=476992]: Done uploading piece. err: <nil> Dec 28 07:24:34.146817494 (543h55m34.017900539s) [noise=true , address=145.79.227.14:4839 , piece_id=HWBXXJKYTV7C6TOBWMK7U5QJVRROMVPLZLPAP7URZ2E6J6TKBOVA, seg_pos={0 0}, upload=476992]: Done uploading piece. err: <nil> Dec 28 07:24:34.146901404 (543h55m34.017984449s) [noise=true , address=145.79.227.31:4798 , piece_id=6MA7SUPGRYZ7RZSIURS3IFEE4B7VRDFWLYVTWCQFHSYMGX5ULP6A, seg_pos={0 0}, upload=476992]: Done uploading piece. err: <nil> Dec 28 07:24:34.154751622 (543h55m34.025834657s) [noise=true , address=145.79.227.18:4007 , piece_id=TZ6KLDXOBKVHULQB3E4WLXRACWPTA6C5JZFWITYFFSXK2D5KOCXA, seg_pos={0 0}, upload=476992]: Done uploading piece. err: <nil> Dec 28 07:24:34.362215111 (543h55m34.233298146s) [noise=true , address=45.91.116.12:4225 , piece_id=FEEGNTEUKADXJA6Q525AUFVFNTDAWYGP5HFEVNVJN6Q4CQN2BB3A, seg_pos={0 0}, upload=476992]: Done uploading piece. err: <nil> Dec 28 07:24:34.372017326 (543h55m34.243100371s) [noise=true , address=45.91.116.13:4156 , piece_id=R7RVEBAT5VGQZHUHRLEPBS7STP7XVA6AY4YSLPRIXJKUVKCFF4VQ, seg_pos={0 0}, upload=476992]: Done uploading piece. err: <nil> Dec 28 07:24:34.565728377 (543h55m34.436811422s) [noise=true , address=45.91.116.10:4084 , piece_id=HVGRMOQXOHAFUNUXY2J42H5OJL6XDEMLO3BLROQEFJ6INUAK4I4A, seg_pos={0 0}, upload=476992]: Done uploading piece. err: <nil> Dec 28 07:24:34.565747725 (543h55m34.43683076s) [seg_pos={0 0}, upload=476992]: Segment reached optimal threshold of 13 pieces. Dec 28 07:24:34.566162044 (543h55m34.437245089s) [noise=true , address=77.90.31.13:4626 , piece_id=7DPB5UBH7ABVQEHEZAJRVY5LTLSFU3D5DIZH2NZLYAKUEHZ4UHDA, seg_pos={0 0}, upload=476992]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu): protocol: expected piece hash; context canceled; EOF Dec 28 07:24:34.566169999 (543h55m34.437253044s) [noise=true , address=77.90.31.13:4626 , piece_id=7DPB5UBH7ABVQEHEZAJRVY5LTLSFU3D5DIZH2NZLYAKUEHZ4UHDA, seg_pos={0 0}, upload=476992]: Long tail cancellation due to optimal threshold Dec 28 07:24:34.566514885 (543h55m34.437597921s) [noise=true , address=45.91.116.5:4297 , piece_id=TRXYL52PWBQI3U3H5S7XFNEVTIKZXJXVUL7B5H3YL7RWQNVAWSDA, seg_pos={0 0}, upload=476992]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: write tcp 188.166.225.170:41944->45.91.116.5:4297: use of closed network connection; write tcp 188.166.225.170:41944->45.91.116.5:4297: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41944->45.91.116.5:4297: use of closed network connection Dec 28 07:24:34.566522660 (543h55m34.437605695s) [noise=true , address=45.91.116.5:4297 , piece_id=TRXYL52PWBQI3U3H5S7XFNEVTIKZXJXVUL7B5H3YL7RWQNVAWSDA, seg_pos={0 0}, upload=476992]: Long tail cancellation due to optimal threshold Dec 28 07:24:34.566883597 (543h55m34.437966632s) [noise=true , address=45.91.116.8:4046 , piece_id=2FECWK73M4LDD3WHQTQY53KETVJY4FHMSDABWBLOLODMZIJZVNFQ, seg_pos={0 0}, upload=476992]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H): protocol: context canceled; context canceled Dec 28 07:24:34.566890451 (543h55m34.437973486s) [noise=true , address=45.91.116.8:4046 , piece_id=2FECWK73M4LDD3WHQTQY53KETVJY4FHMSDABWBLOLODMZIJZVNFQ, seg_pos={0 0}, upload=476992]: Long tail cancellation due to optimal threshold Dec 28 07:24:34.566907002 (543h55m34.437990037s) [seg_pos={0 0}, upload=476992]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 07:24:34.567001582 (543h55m34.438084617s) [upload=476992]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 07:24:34.567006662 (543h55m34.438089697s) [upload=476992]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 07:24:34.862454529 (543h55m34.733537564s) [upload=476992]: Done uploading segments... Dec 28 07:24:34.862480950 (543h55m34.733563985s) [upload=476992]: Upload finished. err: <nil> Duration: 5.804255944s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [X.XX.....X...XXX.XX.............XX.X...X...X.XXX..X.XX.X....] duration: 5.648729645s trace: https://jaeger.dev.storj.io/trace/19034703e3293256 FAIL (2025-12-28 07:27:12 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 28 07:24:31.869041095 (543h55m31.740124131s) [upload=476992]: Starting upload Dec 28 07:24:31.869091793 (543h55m31.740174838s) [upload=476992]: Done starting upload Dec 28 07:24:31.869203846 (543h55m31.740286901s) [upload=476992]: Uploading segments... Dec 28 07:24:31.869341798 (543h55m31.740424854s) [upload=476992]: Got next segment. Inline: false Dec 28 07:24:31.869387505 (543h55m31.74047054s) [upload=476992]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 07:24:31.869392474 (543h55m31.74047551s) [upload=476992]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 07:24:32.112124481 (543h55m31.983207526s) [seg_pos={0 0}, upload=476992]: Begin upload segment... Dec 28 07:24:32.112187041 (543h55m31.983270086s) [seg_pos={0 0}, upload=476992]: Done begin upload segment. Dec 28 07:24:32.112262514 (543h55m31.983345549s) [noise=true , address=145.79.227.45:4231 , piece_id=M4CCTNOTHYYE2YXIGKONI26OH7W2I4R3IQJH2TXY6ZFKLRYTBPNA, seg_pos={0 0}, upload=476992]: Uploading piece... Dec 28 07:24:32.112347206 (543h55m31.983430251s) [noise=true , address=145.79.227.31:4798 , piece_id=6MA7SUPGRYZ7RZSIURS3IFEE4B7VRDFWLYVTWCQFHSYMGX5ULP6A, seg_pos={0 0}, upload=476992]: Uploading piece... Dec 28 07:24:32.112868147 (543h55m31.983951192s) [noise=true , address=145.79.227.7:4293 , piece_id=ILHS7U65IXGQUHGUZOSKDLR75D2ADT6VMY2DF3Q44JGA6NC35YQA, seg_pos={0 0}, upload=476992]: Uploading piece... Dec 28 07:24:32.113275393 (543h55m31.984358438s) [noise=true , address=145.79.227.17:4972 , piece_id=2DEF4NOZF5IEFEHLS3C245RYLN6TKKHIEYAVXJWK2EZDTDB5LZ6A, seg_pos={0 0}, upload=476992]: Uploading piece... Dec 28 07:24:32.114689235 (543h55m31.98577228s) [noise=true , address=45.91.116.5:4297 , piece_id=TRXYL52PWBQI3U3H5S7XFNEVTIKZXJXVUL7B5H3YL7RWQNVAWSDA, seg_pos={0 0}, upload=476992]: Uploading piece... Dec 28 07:24:32.115243370 (543h55m31.986326416s) [noise=true , address=145.79.227.35:4481 , piece_id=45CJ7I6U6OJLKJNNCECWBTRWKL3MDXPAP5TSWR57JIXAQ7QTF5CQ, seg_pos={0 0}, upload=476992]: Uploading piece... Dec 28 07:24:32.115709027 (543h55m31.986792072s) [noise=true , address=45.91.116.8:4046 , piece_id=2FECWK73M4LDD3WHQTQY53KETVJY4FHMSDABWBLOLODMZIJZVNFQ, seg_pos={0 0}, upload=476992]: Uploading piece... Dec 28 07:24:32.116664757 (543h55m31.987747792s) [noise=true , address=45.91.116.12:4225 , piece_id=FEEGNTEUKADXJA6Q525AUFVFNTDAWYGP5HFEVNVJN6Q4CQN2BB3A, seg_pos={0 0}, upload=476992]: Uploading piece... Dec 28 07:24:32.117199886 (543h55m31.988282931s) [noise=true , address=145.79.227.14:4839 , piece_id=HWBXXJKYTV7C6TOBWMK7U5QJVRROMVPLZLPAP7URZ2E6J6TKBOVA, seg_pos={0 0}, upload=476992]: Uploading piece... Dec 28 07:24:32.118394961 (543h55m31.989477996s) [noise=true , address=145.79.227.37:4567 , piece_id=OPWBMOJSFIVAV2ZQVTSPIUU3RWB6R35COP75KWQW7N4EM3EUE3MA, seg_pos={0 0}, upload=476992]: Uploading piece... Dec 28 07:24:32.119110775 (543h55m31.99019382s) [noise=true , address=77.90.31.13:4626 , piece_id=7DPB5UBH7ABVQEHEZAJRVY5LTLSFU3D5DIZH2NZLYAKUEHZ4UHDA, seg_pos={0 0}, upload=476992]: Uploading piece... Dec 28 07:24:32.122486352 (543h55m31.993569397s) [noise=true , address=45.91.116.13:4156 , piece_id=R7RVEBAT5VGQZHUHRLEPBS7STP7XVA6AY4YSLPRIXJKUVKCFF4VQ, seg_pos={0 0}, upload=476992]: Uploading piece... Dec 28 07:24:32.124638781 (543h55m31.995721816s) [noise=true , address=145.79.227.16:4911 , piece_id=HDA2REQUIC2Z7XRKP77UDNOWFVITNSXSMNFQA7NHIVYJNBB6SIZQ, seg_pos={0 0}, upload=476992]: Uploading piece... Dec 28 07:24:32.124849412 (543h55m31.995932457s) [noise=true , address=145.79.227.18:4007 , piece_id=TZ6KLDXOBKVHULQB3E4WLXRACWPTA6C5JZFWITYFFSXK2D5KOCXA, seg_pos={0 0}, upload=476992]: Uploading piece... Dec 28 07:24:32.126406407 (543h55m31.997489442s) [noise=true , address=45.91.116.10:4084 , piece_id=HVGRMOQXOHAFUNUXY2J42H5OJL6XDEMLO3BLROQEFJ6INUAK4I4A, seg_pos={0 0}, upload=476992]: Uploading piece... Dec 28 07:24:32.127679831 (543h55m31.998762886s) [noise=true , address=145.79.227.42:4123 , piece_id=TIHM4TDHHEXOITN4UL6ZTQUGPD3KU67QNH5Y2HZX7IYQ6EBHKXUQ, seg_pos={0 0}, upload=476992]: Uploading piece... Dec 28 07:24:33.749682045 (543h55m33.62076508s) [upload=476992]: Next returned nil segment Dec 28 07:24:33.749699568 (543h55m33.620782613s) [upload=476992]: Waiting for error group managing segments... Dec 28 07:24:34.119458798 (543h55m33.990541843s) [noise=true , address=145.79.227.45:4231 , piece_id=M4CCTNOTHYYE2YXIGKONI26OH7W2I4R3IQJH2TXY6ZFKLRYTBPNA, seg_pos={0 0}, upload=476992]: Done uploading piece. err: <nil> Dec 28 07:24:34.121601728 (543h55m33.992684763s) [noise=true , address=145.79.227.17:4972 , piece_id=2DEF4NOZF5IEFEHLS3C245RYLN6TKKHIEYAVXJWK2EZDTDB5LZ6A, seg_pos={0 0}, upload=476992]: Done uploading piece. err: <nil> Dec 28 07:24:34.128325031 (543h55m33.999408066s) [noise=true , address=145.79.227.37:4567 , piece_id=OPWBMOJSFIVAV2ZQVTSPIUU3RWB6R35COP75KWQW7N4EM3EUE3MA, seg_pos={0 0}, upload=476992]: Done uploading piece. err: <nil> Dec 28 07:24:34.128912158 (543h55m33.999995203s) [noise=true , address=145.79.227.16:4911 , piece_id=HDA2REQUIC2Z7XRKP77UDNOWFVITNSXSMNFQA7NHIVYJNBB6SIZQ, seg_pos={0 0}, upload=476992]: Done uploading piece. err: <nil> Dec 28 07:24:34.131137215 (543h55m34.00222025s) [noise=true , address=145.79.227.42:4123 , piece_id=TIHM4TDHHEXOITN4UL6ZTQUGPD3KU67QNH5Y2HZX7IYQ6EBHKXUQ, seg_pos={0 0}, upload=476992]: Done uploading piece. err: <nil> Dec 28 07:24:34.133603162 (543h55m34.004686197s) [noise=true , address=145.79.227.7:4293 , piece_id=ILHS7U65IXGQUHGUZOSKDLR75D2ADT6VMY2DF3Q44JGA6NC35YQA, seg_pos={0 0}, upload=476992]: Done uploading piece. err: <nil> Dec 28 07:24:34.140904865 (543h55m34.0119879s) [noise=true , address=145.79.227.35:4481 , piece_id=45CJ7I6U6OJLKJNNCECWBTRWKL3MDXPAP5TSWR57JIXAQ7QTF5CQ, seg_pos={0 0}, upload=476992]: Done uploading piece. err: <nil> Dec 28 07:24:34.146817494 (543h55m34.017900539s) [noise=true , address=145.79.227.14:4839 , piece_id=HWBXXJKYTV7C6TOBWMK7U5QJVRROMVPLZLPAP7URZ2E6J6TKBOVA, seg_pos={0 0}, upload=476992]: Done uploading piece. err: <nil> Dec 28 07:24:34.146901404 (543h55m34.017984449s) [noise=true , address=145.79.227.31:4798 , piece_id=6MA7SUPGRYZ7RZSIURS3IFEE4B7VRDFWLYVTWCQFHSYMGX5ULP6A, seg_pos={0 0}, upload=476992]: Done uploading piece. err: <nil> Dec 28 07:24:34.154751622 (543h55m34.025834657s) [noise=true , address=145.79.227.18:4007 , piece_id=TZ6KLDXOBKVHULQB3E4WLXRACWPTA6C5JZFWITYFFSXK2D5KOCXA, seg_pos={0 0}, upload=476992]: Done uploading piece. err: <nil> Dec 28 07:24:34.362215111 (543h55m34.233298146s) [noise=true , address=45.91.116.12:4225 , piece_id=FEEGNTEUKADXJA6Q525AUFVFNTDAWYGP5HFEVNVJN6Q4CQN2BB3A, seg_pos={0 0}, upload=476992]: Done uploading piece. err: <nil> Dec 28 07:24:34.372017326 (543h55m34.243100371s) [noise=true , address=45.91.116.13:4156 , piece_id=R7RVEBAT5VGQZHUHRLEPBS7STP7XVA6AY4YSLPRIXJKUVKCFF4VQ, seg_pos={0 0}, upload=476992]: Done uploading piece. err: <nil> Dec 28 07:24:34.565728377 (543h55m34.436811422s) [noise=true , address=45.91.116.10:4084 , piece_id=HVGRMOQXOHAFUNUXY2J42H5OJL6XDEMLO3BLROQEFJ6INUAK4I4A, seg_pos={0 0}, upload=476992]: Done uploading piece. err: <nil> Dec 28 07:24:34.565747725 (543h55m34.43683076s) [seg_pos={0 0}, upload=476992]: Segment reached optimal threshold of 13 pieces. Dec 28 07:24:34.566162044 (543h55m34.437245089s) [noise=true , address=77.90.31.13:4626 , piece_id=7DPB5UBH7ABVQEHEZAJRVY5LTLSFU3D5DIZH2NZLYAKUEHZ4UHDA, seg_pos={0 0}, upload=476992]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu): protocol: expected piece hash; context canceled; EOF Dec 28 07:24:34.566169999 (543h55m34.437253044s) [noise=true , address=77.90.31.13:4626 , piece_id=7DPB5UBH7ABVQEHEZAJRVY5LTLSFU3D5DIZH2NZLYAKUEHZ4UHDA, seg_pos={0 0}, upload=476992]: Long tail cancellation due to optimal threshold Dec 28 07:24:34.566514885 (543h55m34.437597921s) [noise=true , address=45.91.116.5:4297 , piece_id=TRXYL52PWBQI3U3H5S7XFNEVTIKZXJXVUL7B5H3YL7RWQNVAWSDA, seg_pos={0 0}, upload=476992]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: write tcp 188.166.225.170:41944->45.91.116.5:4297: use of closed network connection; write tcp 188.166.225.170:41944->45.91.116.5:4297: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41944->45.91.116.5:4297: use of closed network connection Dec 28 07:24:34.566522660 (543h55m34.437605695s) [noise=true , address=45.91.116.5:4297 , piece_id=TRXYL52PWBQI3U3H5S7XFNEVTIKZXJXVUL7B5H3YL7RWQNVAWSDA, seg_pos={0 0}, upload=476992]: Long tail cancellation due to optimal threshold Dec 28 07:24:34.566883597 (543h55m34.437966632s) [noise=true , address=45.91.116.8:4046 , piece_id=2FECWK73M4LDD3WHQTQY53KETVJY4FHMSDABWBLOLODMZIJZVNFQ, seg_pos={0 0}, upload=476992]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H): protocol: context canceled; context canceled Dec 28 07:24:34.566890451 (543h55m34.437973486s) [noise=true , address=45.91.116.8:4046 , piece_id=2FECWK73M4LDD3WHQTQY53KETVJY4FHMSDABWBLOLODMZIJZVNFQ, seg_pos={0 0}, upload=476992]: Long tail cancellation due to optimal threshold Dec 28 07:24:34.566907002 (543h55m34.437990037s) [seg_pos={0 0}, upload=476992]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 07:24:34.567001582 (543h55m34.438084617s) [upload=476992]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 07:24:34.567006662 (543h55m34.438089697s) [upload=476992]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 07:24:34.862454529 (543h55m34.733537564s) [upload=476992]: Done uploading segments... Dec 28 07:24:34.862480950 (543h55m34.733563985s) [upload=476992]: Upload finished. err: <nil> Duration: 5.648766725s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 25 out of 60 is failed (threshold=6) [....X.X...XX......X.X..X..XX.XXXXXXX.X...X..X.XX.....X.X.XX.] duration: 5.000760868s trace: https://jaeger.dev.storj.io/trace/56bbf53c3df5a92f FAIL (2025-12-28 07:27:01 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000785865s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [XX.XX.....X...XXX.XX.............XX.X...X...X.XXX..X.XX.X...] duration: 5.991591133s trace: https://jaeger.dev.storj.io/trace/19034703e3293256 FAIL (2025-12-28 07:26:43 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 28 07:24:31.869041095 (543h55m31.740124131s) [upload=476992]: Starting upload Dec 28 07:24:31.869091793 (543h55m31.740174838s) [upload=476992]: Done starting upload Dec 28 07:24:31.869203846 (543h55m31.740286901s) [upload=476992]: Uploading segments... Dec 28 07:24:31.869341798 (543h55m31.740424854s) [upload=476992]: Got next segment. Inline: false Dec 28 07:24:31.869387505 (543h55m31.74047054s) [upload=476992]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 07:24:31.869392474 (543h55m31.74047551s) [upload=476992]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 07:24:32.112124481 (543h55m31.983207526s) [seg_pos={0 0}, upload=476992]: Begin upload segment... Dec 28 07:24:32.112187041 (543h55m31.983270086s) [seg_pos={0 0}, upload=476992]: Done begin upload segment. Dec 28 07:24:32.112262514 (543h55m31.983345549s) [noise=true , address=145.79.227.45:4231 , piece_id=M4CCTNOTHYYE2YXIGKONI26OH7W2I4R3IQJH2TXY6ZFKLRYTBPNA, seg_pos={0 0}, upload=476992]: Uploading piece... Dec 28 07:24:32.112347206 (543h55m31.983430251s) [noise=true , address=145.79.227.31:4798 , piece_id=6MA7SUPGRYZ7RZSIURS3IFEE4B7VRDFWLYVTWCQFHSYMGX5ULP6A, seg_pos={0 0}, upload=476992]: Uploading piece... Dec 28 07:24:32.112868147 (543h55m31.983951192s) [noise=true , address=145.79.227.7:4293 , piece_id=ILHS7U65IXGQUHGUZOSKDLR75D2ADT6VMY2DF3Q44JGA6NC35YQA, seg_pos={0 0}, upload=476992]: Uploading piece... Dec 28 07:24:32.113275393 (543h55m31.984358438s) [noise=true , address=145.79.227.17:4972 , piece_id=2DEF4NOZF5IEFEHLS3C245RYLN6TKKHIEYAVXJWK2EZDTDB5LZ6A, seg_pos={0 0}, upload=476992]: Uploading piece... Dec 28 07:24:32.114689235 (543h55m31.98577228s) [noise=true , address=45.91.116.5:4297 , piece_id=TRXYL52PWBQI3U3H5S7XFNEVTIKZXJXVUL7B5H3YL7RWQNVAWSDA, seg_pos={0 0}, upload=476992]: Uploading piece... Dec 28 07:24:32.115243370 (543h55m31.986326416s) [noise=true , address=145.79.227.35:4481 , piece_id=45CJ7I6U6OJLKJNNCECWBTRWKL3MDXPAP5TSWR57JIXAQ7QTF5CQ, seg_pos={0 0}, upload=476992]: Uploading piece... Dec 28 07:24:32.115709027 (543h55m31.986792072s) [noise=true , address=45.91.116.8:4046 , piece_id=2FECWK73M4LDD3WHQTQY53KETVJY4FHMSDABWBLOLODMZIJZVNFQ, seg_pos={0 0}, upload=476992]: Uploading piece... Dec 28 07:24:32.116664757 (543h55m31.987747792s) [noise=true , address=45.91.116.12:4225 , piece_id=FEEGNTEUKADXJA6Q525AUFVFNTDAWYGP5HFEVNVJN6Q4CQN2BB3A, seg_pos={0 0}, upload=476992]: Uploading piece... Dec 28 07:24:32.117199886 (543h55m31.988282931s) [noise=true , address=145.79.227.14:4839 , piece_id=HWBXXJKYTV7C6TOBWMK7U5QJVRROMVPLZLPAP7URZ2E6J6TKBOVA, seg_pos={0 0}, upload=476992]: Uploading piece... Dec 28 07:24:32.118394961 (543h55m31.989477996s) [noise=true , address=145.79.227.37:4567 , piece_id=OPWBMOJSFIVAV2ZQVTSPIUU3RWB6R35COP75KWQW7N4EM3EUE3MA, seg_pos={0 0}, upload=476992]: Uploading piece... Dec 28 07:24:32.119110775 (543h55m31.99019382s) [noise=true , address=77.90.31.13:4626 , piece_id=7DPB5UBH7ABVQEHEZAJRVY5LTLSFU3D5DIZH2NZLYAKUEHZ4UHDA, seg_pos={0 0}, upload=476992]: Uploading piece... Dec 28 07:24:32.122486352 (543h55m31.993569397s) [noise=true , address=45.91.116.13:4156 , piece_id=R7RVEBAT5VGQZHUHRLEPBS7STP7XVA6AY4YSLPRIXJKUVKCFF4VQ, seg_pos={0 0}, upload=476992]: Uploading piece... Dec 28 07:24:32.124638781 (543h55m31.995721816s) [noise=true , address=145.79.227.16:4911 , piece_id=HDA2REQUIC2Z7XRKP77UDNOWFVITNSXSMNFQA7NHIVYJNBB6SIZQ, seg_pos={0 0}, upload=476992]: Uploading piece... Dec 28 07:24:32.124849412 (543h55m31.995932457s) [noise=true , address=145.79.227.18:4007 , piece_id=TZ6KLDXOBKVHULQB3E4WLXRACWPTA6C5JZFWITYFFSXK2D5KOCXA, seg_pos={0 0}, upload=476992]: Uploading piece... Dec 28 07:24:32.126406407 (543h55m31.997489442s) [noise=true , address=45.91.116.10:4084 , piece_id=HVGRMOQXOHAFUNUXY2J42H5OJL6XDEMLO3BLROQEFJ6INUAK4I4A, seg_pos={0 0}, upload=476992]: Uploading piece... Dec 28 07:24:32.127679831 (543h55m31.998762886s) [noise=true , address=145.79.227.42:4123 , piece_id=TIHM4TDHHEXOITN4UL6ZTQUGPD3KU67QNH5Y2HZX7IYQ6EBHKXUQ, seg_pos={0 0}, upload=476992]: Uploading piece... Dec 28 07:24:33.749682045 (543h55m33.62076508s) [upload=476992]: Next returned nil segment Dec 28 07:24:33.749699568 (543h55m33.620782613s) [upload=476992]: Waiting for error group managing segments... Dec 28 07:24:34.119458798 (543h55m33.990541843s) [noise=true , address=145.79.227.45:4231 , piece_id=M4CCTNOTHYYE2YXIGKONI26OH7W2I4R3IQJH2TXY6ZFKLRYTBPNA, seg_pos={0 0}, upload=476992]: Done uploading piece. err: <nil> Dec 28 07:24:34.121601728 (543h55m33.992684763s) [noise=true , address=145.79.227.17:4972 , piece_id=2DEF4NOZF5IEFEHLS3C245RYLN6TKKHIEYAVXJWK2EZDTDB5LZ6A, seg_pos={0 0}, upload=476992]: Done uploading piece. err: <nil> Dec 28 07:24:34.128325031 (543h55m33.999408066s) [noise=true , address=145.79.227.37:4567 , piece_id=OPWBMOJSFIVAV2ZQVTSPIUU3RWB6R35COP75KWQW7N4EM3EUE3MA, seg_pos={0 0}, upload=476992]: Done uploading piece. err: <nil> Dec 28 07:24:34.128912158 (543h55m33.999995203s) [noise=true , address=145.79.227.16:4911 , piece_id=HDA2REQUIC2Z7XRKP77UDNOWFVITNSXSMNFQA7NHIVYJNBB6SIZQ, seg_pos={0 0}, upload=476992]: Done uploading piece. err: <nil> Dec 28 07:24:34.131137215 (543h55m34.00222025s) [noise=true , address=145.79.227.42:4123 , piece_id=TIHM4TDHHEXOITN4UL6ZTQUGPD3KU67QNH5Y2HZX7IYQ6EBHKXUQ, seg_pos={0 0}, upload=476992]: Done uploading piece. err: <nil> Dec 28 07:24:34.133603162 (543h55m34.004686197s) [noise=true , address=145.79.227.7:4293 , piece_id=ILHS7U65IXGQUHGUZOSKDLR75D2ADT6VMY2DF3Q44JGA6NC35YQA, seg_pos={0 0}, upload=476992]: Done uploading piece. err: <nil> Dec 28 07:24:34.140904865 (543h55m34.0119879s) [noise=true , address=145.79.227.35:4481 , piece_id=45CJ7I6U6OJLKJNNCECWBTRWKL3MDXPAP5TSWR57JIXAQ7QTF5CQ, seg_pos={0 0}, upload=476992]: Done uploading piece. err: <nil> Dec 28 07:24:34.146817494 (543h55m34.017900539s) [noise=true , address=145.79.227.14:4839 , piece_id=HWBXXJKYTV7C6TOBWMK7U5QJVRROMVPLZLPAP7URZ2E6J6TKBOVA, seg_pos={0 0}, upload=476992]: Done uploading piece. err: <nil> Dec 28 07:24:34.146901404 (543h55m34.017984449s) [noise=true , address=145.79.227.31:4798 , piece_id=6MA7SUPGRYZ7RZSIURS3IFEE4B7VRDFWLYVTWCQFHSYMGX5ULP6A, seg_pos={0 0}, upload=476992]: Done uploading piece. err: <nil> Dec 28 07:24:34.154751622 (543h55m34.025834657s) [noise=true , address=145.79.227.18:4007 , piece_id=TZ6KLDXOBKVHULQB3E4WLXRACWPTA6C5JZFWITYFFSXK2D5KOCXA, seg_pos={0 0}, upload=476992]: Done uploading piece. err: <nil> Dec 28 07:24:34.362215111 (543h55m34.233298146s) [noise=true , address=45.91.116.12:4225 , piece_id=FEEGNTEUKADXJA6Q525AUFVFNTDAWYGP5HFEVNVJN6Q4CQN2BB3A, seg_pos={0 0}, upload=476992]: Done uploading piece. err: <nil> Dec 28 07:24:34.372017326 (543h55m34.243100371s) [noise=true , address=45.91.116.13:4156 , piece_id=R7RVEBAT5VGQZHUHRLEPBS7STP7XVA6AY4YSLPRIXJKUVKCFF4VQ, seg_pos={0 0}, upload=476992]: Done uploading piece. err: <nil> Dec 28 07:24:34.565728377 (543h55m34.436811422s) [noise=true , address=45.91.116.10:4084 , piece_id=HVGRMOQXOHAFUNUXY2J42H5OJL6XDEMLO3BLROQEFJ6INUAK4I4A, seg_pos={0 0}, upload=476992]: Done uploading piece. err: <nil> Dec 28 07:24:34.565747725 (543h55m34.43683076s) [seg_pos={0 0}, upload=476992]: Segment reached optimal threshold of 13 pieces. Dec 28 07:24:34.566162044 (543h55m34.437245089s) [noise=true , address=77.90.31.13:4626 , piece_id=7DPB5UBH7ABVQEHEZAJRVY5LTLSFU3D5DIZH2NZLYAKUEHZ4UHDA, seg_pos={0 0}, upload=476992]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu): protocol: expected piece hash; context canceled; EOF Dec 28 07:24:34.566169999 (543h55m34.437253044s) [noise=true , address=77.90.31.13:4626 , piece_id=7DPB5UBH7ABVQEHEZAJRVY5LTLSFU3D5DIZH2NZLYAKUEHZ4UHDA, seg_pos={0 0}, upload=476992]: Long tail cancellation due to optimal threshold Dec 28 07:24:34.566514885 (543h55m34.437597921s) [noise=true , address=45.91.116.5:4297 , piece_id=TRXYL52PWBQI3U3H5S7XFNEVTIKZXJXVUL7B5H3YL7RWQNVAWSDA, seg_pos={0 0}, upload=476992]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: write tcp 188.166.225.170:41944->45.91.116.5:4297: use of closed network connection; write tcp 188.166.225.170:41944->45.91.116.5:4297: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41944->45.91.116.5:4297: use of closed network connection Dec 28 07:24:34.566522660 (543h55m34.437605695s) [noise=true , address=45.91.116.5:4297 , piece_id=TRXYL52PWBQI3U3H5S7XFNEVTIKZXJXVUL7B5H3YL7RWQNVAWSDA, seg_pos={0 0}, upload=476992]: Long tail cancellation due to optimal threshold Dec 28 07:24:34.566883597 (543h55m34.437966632s) [noise=true , address=45.91.116.8:4046 , piece_id=2FECWK73M4LDD3WHQTQY53KETVJY4FHMSDABWBLOLODMZIJZVNFQ, seg_pos={0 0}, upload=476992]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H): protocol: context canceled; context canceled Dec 28 07:24:34.566890451 (543h55m34.437973486s) [noise=true , address=45.91.116.8:4046 , piece_id=2FECWK73M4LDD3WHQTQY53KETVJY4FHMSDABWBLOLODMZIJZVNFQ, seg_pos={0 0}, upload=476992]: Long tail cancellation due to optimal threshold Dec 28 07:24:34.566907002 (543h55m34.437990037s) [seg_pos={0 0}, upload=476992]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 07:24:34.567001582 (543h55m34.438084617s) [upload=476992]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 07:24:34.567006662 (543h55m34.438089697s) [upload=476992]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 07:24:34.862454529 (543h55m34.733537564s) [upload=476992]: Done uploading segments... Dec 28 07:24:34.862480950 (543h55m34.733563985s) [upload=476992]: Upload finished. err: <nil> Duration: 5.991618525s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 23 out of 60 is failed (threshold=3) [XXX.XX.....X...XXX.XX.............XX.X...X...X.XXX..X.XX.X..] duration: 5.992335149s trace: https://jaeger.dev.storj.io/trace/19034703e3293256 FAIL (2025-12-28 07:26:17 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 28 07:24:31.869041095 (543h55m31.740124131s) [upload=476992]: Starting upload Dec 28 07:24:31.869091793 (543h55m31.740174838s) [upload=476992]: Done starting upload Dec 28 07:24:31.869203846 (543h55m31.740286901s) [upload=476992]: Uploading segments... Dec 28 07:24:31.869341798 (543h55m31.740424854s) [upload=476992]: Got next segment. Inline: false Dec 28 07:24:31.869387505 (543h55m31.74047054s) [upload=476992]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 07:24:31.869392474 (543h55m31.74047551s) [upload=476992]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 07:24:32.112124481 (543h55m31.983207526s) [seg_pos={0 0}, upload=476992]: Begin upload segment... Dec 28 07:24:32.112187041 (543h55m31.983270086s) [seg_pos={0 0}, upload=476992]: Done begin upload segment. Dec 28 07:24:32.112262514 (543h55m31.983345549s) [noise=true , address=145.79.227.45:4231 , piece_id=M4CCTNOTHYYE2YXIGKONI26OH7W2I4R3IQJH2TXY6ZFKLRYTBPNA, seg_pos={0 0}, upload=476992]: Uploading piece... Dec 28 07:24:32.112347206 (543h55m31.983430251s) [noise=true , address=145.79.227.31:4798 , piece_id=6MA7SUPGRYZ7RZSIURS3IFEE4B7VRDFWLYVTWCQFHSYMGX5ULP6A, seg_pos={0 0}, upload=476992]: Uploading piece... Dec 28 07:24:32.112868147 (543h55m31.983951192s) [noise=true , address=145.79.227.7:4293 , piece_id=ILHS7U65IXGQUHGUZOSKDLR75D2ADT6VMY2DF3Q44JGA6NC35YQA, seg_pos={0 0}, upload=476992]: Uploading piece... Dec 28 07:24:32.113275393 (543h55m31.984358438s) [noise=true , address=145.79.227.17:4972 , piece_id=2DEF4NOZF5IEFEHLS3C245RYLN6TKKHIEYAVXJWK2EZDTDB5LZ6A, seg_pos={0 0}, upload=476992]: Uploading piece... Dec 28 07:24:32.114689235 (543h55m31.98577228s) [noise=true , address=45.91.116.5:4297 , piece_id=TRXYL52PWBQI3U3H5S7XFNEVTIKZXJXVUL7B5H3YL7RWQNVAWSDA, seg_pos={0 0}, upload=476992]: Uploading piece... Dec 28 07:24:32.115243370 (543h55m31.986326416s) [noise=true , address=145.79.227.35:4481 , piece_id=45CJ7I6U6OJLKJNNCECWBTRWKL3MDXPAP5TSWR57JIXAQ7QTF5CQ, seg_pos={0 0}, upload=476992]: Uploading piece... Dec 28 07:24:32.115709027 (543h55m31.986792072s) [noise=true , address=45.91.116.8:4046 , piece_id=2FECWK73M4LDD3WHQTQY53KETVJY4FHMSDABWBLOLODMZIJZVNFQ, seg_pos={0 0}, upload=476992]: Uploading piece... Dec 28 07:24:32.116664757 (543h55m31.987747792s) [noise=true , address=45.91.116.12:4225 , piece_id=FEEGNTEUKADXJA6Q525AUFVFNTDAWYGP5HFEVNVJN6Q4CQN2BB3A, seg_pos={0 0}, upload=476992]: Uploading piece... Dec 28 07:24:32.117199886 (543h55m31.988282931s) [noise=true , address=145.79.227.14:4839 , piece_id=HWBXXJKYTV7C6TOBWMK7U5QJVRROMVPLZLPAP7URZ2E6J6TKBOVA, seg_pos={0 0}, upload=476992]: Uploading piece... Dec 28 07:24:32.118394961 (543h55m31.989477996s) [noise=true , address=145.79.227.37:4567 , piece_id=OPWBMOJSFIVAV2ZQVTSPIUU3RWB6R35COP75KWQW7N4EM3EUE3MA, seg_pos={0 0}, upload=476992]: Uploading piece... Dec 28 07:24:32.119110775 (543h55m31.99019382s) [noise=true , address=77.90.31.13:4626 , piece_id=7DPB5UBH7ABVQEHEZAJRVY5LTLSFU3D5DIZH2NZLYAKUEHZ4UHDA, seg_pos={0 0}, upload=476992]: Uploading piece... Dec 28 07:24:32.122486352 (543h55m31.993569397s) [noise=true , address=45.91.116.13:4156 , piece_id=R7RVEBAT5VGQZHUHRLEPBS7STP7XVA6AY4YSLPRIXJKUVKCFF4VQ, seg_pos={0 0}, upload=476992]: Uploading piece... Dec 28 07:24:32.124638781 (543h55m31.995721816s) [noise=true , address=145.79.227.16:4911 , piece_id=HDA2REQUIC2Z7XRKP77UDNOWFVITNSXSMNFQA7NHIVYJNBB6SIZQ, seg_pos={0 0}, upload=476992]: Uploading piece... Dec 28 07:24:32.124849412 (543h55m31.995932457s) [noise=true , address=145.79.227.18:4007 , piece_id=TZ6KLDXOBKVHULQB3E4WLXRACWPTA6C5JZFWITYFFSXK2D5KOCXA, seg_pos={0 0}, upload=476992]: Uploading piece... Dec 28 07:24:32.126406407 (543h55m31.997489442s) [noise=true , address=45.91.116.10:4084 , piece_id=HVGRMOQXOHAFUNUXY2J42H5OJL6XDEMLO3BLROQEFJ6INUAK4I4A, seg_pos={0 0}, upload=476992]: Uploading piece... Dec 28 07:24:32.127679831 (543h55m31.998762886s) [noise=true , address=145.79.227.42:4123 , piece_id=TIHM4TDHHEXOITN4UL6ZTQUGPD3KU67QNH5Y2HZX7IYQ6EBHKXUQ, seg_pos={0 0}, upload=476992]: Uploading piece... Dec 28 07:24:33.749682045 (543h55m33.62076508s) [upload=476992]: Next returned nil segment Dec 28 07:24:33.749699568 (543h55m33.620782613s) [upload=476992]: Waiting for error group managing segments... Dec 28 07:24:34.119458798 (543h55m33.990541843s) [noise=true , address=145.79.227.45:4231 , piece_id=M4CCTNOTHYYE2YXIGKONI26OH7W2I4R3IQJH2TXY6ZFKLRYTBPNA, seg_pos={0 0}, upload=476992]: Done uploading piece. err: <nil> Dec 28 07:24:34.121601728 (543h55m33.992684763s) [noise=true , address=145.79.227.17:4972 , piece_id=2DEF4NOZF5IEFEHLS3C245RYLN6TKKHIEYAVXJWK2EZDTDB5LZ6A, seg_pos={0 0}, upload=476992]: Done uploading piece. err: <nil> Dec 28 07:24:34.128325031 (543h55m33.999408066s) [noise=true , address=145.79.227.37:4567 , piece_id=OPWBMOJSFIVAV2ZQVTSPIUU3RWB6R35COP75KWQW7N4EM3EUE3MA, seg_pos={0 0}, upload=476992]: Done uploading piece. err: <nil> Dec 28 07:24:34.128912158 (543h55m33.999995203s) [noise=true , address=145.79.227.16:4911 , piece_id=HDA2REQUIC2Z7XRKP77UDNOWFVITNSXSMNFQA7NHIVYJNBB6SIZQ, seg_pos={0 0}, upload=476992]: Done uploading piece. err: <nil> Dec 28 07:24:34.131137215 (543h55m34.00222025s) [noise=true , address=145.79.227.42:4123 , piece_id=TIHM4TDHHEXOITN4UL6ZTQUGPD3KU67QNH5Y2HZX7IYQ6EBHKXUQ, seg_pos={0 0}, upload=476992]: Done uploading piece. err: <nil> Dec 28 07:24:34.133603162 (543h55m34.004686197s) [noise=true , address=145.79.227.7:4293 , piece_id=ILHS7U65IXGQUHGUZOSKDLR75D2ADT6VMY2DF3Q44JGA6NC35YQA, seg_pos={0 0}, upload=476992]: Done uploading piece. err: <nil> Dec 28 07:24:34.140904865 (543h55m34.0119879s) [noise=true , address=145.79.227.35:4481 , piece_id=45CJ7I6U6OJLKJNNCECWBTRWKL3MDXPAP5TSWR57JIXAQ7QTF5CQ, seg_pos={0 0}, upload=476992]: Done uploading piece. err: <nil> Dec 28 07:24:34.146817494 (543h55m34.017900539s) [noise=true , address=145.79.227.14:4839 , piece_id=HWBXXJKYTV7C6TOBWMK7U5QJVRROMVPLZLPAP7URZ2E6J6TKBOVA, seg_pos={0 0}, upload=476992]: Done uploading piece. err: <nil> Dec 28 07:24:34.146901404 (543h55m34.017984449s) [noise=true , address=145.79.227.31:4798 , piece_id=6MA7SUPGRYZ7RZSIURS3IFEE4B7VRDFWLYVTWCQFHSYMGX5ULP6A, seg_pos={0 0}, upload=476992]: Done uploading piece. err: <nil> Dec 28 07:24:34.154751622 (543h55m34.025834657s) [noise=true , address=145.79.227.18:4007 , piece_id=TZ6KLDXOBKVHULQB3E4WLXRACWPTA6C5JZFWITYFFSXK2D5KOCXA, seg_pos={0 0}, upload=476992]: Done uploading piece. err: <nil> Dec 28 07:24:34.362215111 (543h55m34.233298146s) [noise=true , address=45.91.116.12:4225 , piece_id=FEEGNTEUKADXJA6Q525AUFVFNTDAWYGP5HFEVNVJN6Q4CQN2BB3A, seg_pos={0 0}, upload=476992]: Done uploading piece. err: <nil> Dec 28 07:24:34.372017326 (543h55m34.243100371s) [noise=true , address=45.91.116.13:4156 , piece_id=R7RVEBAT5VGQZHUHRLEPBS7STP7XVA6AY4YSLPRIXJKUVKCFF4VQ, seg_pos={0 0}, upload=476992]: Done uploading piece. err: <nil> Dec 28 07:24:34.565728377 (543h55m34.436811422s) [noise=true , address=45.91.116.10:4084 , piece_id=HVGRMOQXOHAFUNUXY2J42H5OJL6XDEMLO3BLROQEFJ6INUAK4I4A, seg_pos={0 0}, upload=476992]: Done uploading piece. err: <nil> Dec 28 07:24:34.565747725 (543h55m34.43683076s) [seg_pos={0 0}, upload=476992]: Segment reached optimal threshold of 13 pieces. Dec 28 07:24:34.566162044 (543h55m34.437245089s) [noise=true , address=77.90.31.13:4626 , piece_id=7DPB5UBH7ABVQEHEZAJRVY5LTLSFU3D5DIZH2NZLYAKUEHZ4UHDA, seg_pos={0 0}, upload=476992]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu): protocol: expected piece hash; context canceled; EOF Dec 28 07:24:34.566169999 (543h55m34.437253044s) [noise=true , address=77.90.31.13:4626 , piece_id=7DPB5UBH7ABVQEHEZAJRVY5LTLSFU3D5DIZH2NZLYAKUEHZ4UHDA, seg_pos={0 0}, upload=476992]: Long tail cancellation due to optimal threshold Dec 28 07:24:34.566514885 (543h55m34.437597921s) [noise=true , address=45.91.116.5:4297 , piece_id=TRXYL52PWBQI3U3H5S7XFNEVTIKZXJXVUL7B5H3YL7RWQNVAWSDA, seg_pos={0 0}, upload=476992]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: write tcp 188.166.225.170:41944->45.91.116.5:4297: use of closed network connection; write tcp 188.166.225.170:41944->45.91.116.5:4297: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41944->45.91.116.5:4297: use of closed network connection Dec 28 07:24:34.566522660 (543h55m34.437605695s) [noise=true , address=45.91.116.5:4297 , piece_id=TRXYL52PWBQI3U3H5S7XFNEVTIKZXJXVUL7B5H3YL7RWQNVAWSDA, seg_pos={0 0}, upload=476992]: Long tail cancellation due to optimal threshold Dec 28 07:24:34.566883597 (543h55m34.437966632s) [noise=true , address=45.91.116.8:4046 , piece_id=2FECWK73M4LDD3WHQTQY53KETVJY4FHMSDABWBLOLODMZIJZVNFQ, seg_pos={0 0}, upload=476992]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H): protocol: context canceled; context canceled Dec 28 07:24:34.566890451 (543h55m34.437973486s) [noise=true , address=45.91.116.8:4046 , piece_id=2FECWK73M4LDD3WHQTQY53KETVJY4FHMSDABWBLOLODMZIJZVNFQ, seg_pos={0 0}, upload=476992]: Long tail cancellation due to optimal threshold Dec 28 07:24:34.566907002 (543h55m34.437990037s) [seg_pos={0 0}, upload=476992]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 07:24:34.567001582 (543h55m34.438084617s) [upload=476992]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 07:24:34.567006662 (543h55m34.438089697s) [upload=476992]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 07:24:34.862454529 (543h55m34.733537564s) [upload=476992]: Done uploading segments... Dec 28 07:24:34.862480950 (543h55m34.733563985s) [upload=476992]: Upload finished. err: <nil> Duration: 5.99236211s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 25 out of 60 is failed (threshold=6) [.....X.X...XX......X.X..X..XX.XXXXXXX.X...X..X.XX.....X.X.XX] duration: 5.001232121s trace: https://jaeger.dev.storj.io/trace/56bbf53c3df5a92f FAIL (2025-12-28 07:26:03 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001258482s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 23 out of 60 is failed (threshold=3) [.XXX.XX.....X...XXX.XX.............XX.X...X...X.XXX..X.XX.X.] duration: 6.016646669s trace: https://jaeger.dev.storj.io/trace/19034703e3293256 FAIL (2025-12-28 07:25:45 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 28 07:24:31.869041095 (543h55m31.740124131s) [upload=476992]: Starting upload Dec 28 07:24:31.869091793 (543h55m31.740174838s) [upload=476992]: Done starting upload Dec 28 07:24:31.869203846 (543h55m31.740286901s) [upload=476992]: Uploading segments... Dec 28 07:24:31.869341798 (543h55m31.740424854s) [upload=476992]: Got next segment. Inline: false Dec 28 07:24:31.869387505 (543h55m31.74047054s) [upload=476992]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 07:24:31.869392474 (543h55m31.74047551s) [upload=476992]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 07:24:32.112124481 (543h55m31.983207526s) [seg_pos={0 0}, upload=476992]: Begin upload segment... Dec 28 07:24:32.112187041 (543h55m31.983270086s) [seg_pos={0 0}, upload=476992]: Done begin upload segment. Dec 28 07:24:32.112262514 (543h55m31.983345549s) [noise=true , address=145.79.227.45:4231 , piece_id=M4CCTNOTHYYE2YXIGKONI26OH7W2I4R3IQJH2TXY6ZFKLRYTBPNA, seg_pos={0 0}, upload=476992]: Uploading piece... Dec 28 07:24:32.112347206 (543h55m31.983430251s) [noise=true , address=145.79.227.31:4798 , piece_id=6MA7SUPGRYZ7RZSIURS3IFEE4B7VRDFWLYVTWCQFHSYMGX5ULP6A, seg_pos={0 0}, upload=476992]: Uploading piece... Dec 28 07:24:32.112868147 (543h55m31.983951192s) [noise=true , address=145.79.227.7:4293 , piece_id=ILHS7U65IXGQUHGUZOSKDLR75D2ADT6VMY2DF3Q44JGA6NC35YQA, seg_pos={0 0}, upload=476992]: Uploading piece... Dec 28 07:24:32.113275393 (543h55m31.984358438s) [noise=true , address=145.79.227.17:4972 , piece_id=2DEF4NOZF5IEFEHLS3C245RYLN6TKKHIEYAVXJWK2EZDTDB5LZ6A, seg_pos={0 0}, upload=476992]: Uploading piece... Dec 28 07:24:32.114689235 (543h55m31.98577228s) [noise=true , address=45.91.116.5:4297 , piece_id=TRXYL52PWBQI3U3H5S7XFNEVTIKZXJXVUL7B5H3YL7RWQNVAWSDA, seg_pos={0 0}, upload=476992]: Uploading piece... Dec 28 07:24:32.115243370 (543h55m31.986326416s) [noise=true , address=145.79.227.35:4481 , piece_id=45CJ7I6U6OJLKJNNCECWBTRWKL3MDXPAP5TSWR57JIXAQ7QTF5CQ, seg_pos={0 0}, upload=476992]: Uploading piece... Dec 28 07:24:32.115709027 (543h55m31.986792072s) [noise=true , address=45.91.116.8:4046 , piece_id=2FECWK73M4LDD3WHQTQY53KETVJY4FHMSDABWBLOLODMZIJZVNFQ, seg_pos={0 0}, upload=476992]: Uploading piece... Dec 28 07:24:32.116664757 (543h55m31.987747792s) [noise=true , address=45.91.116.12:4225 , piece_id=FEEGNTEUKADXJA6Q525AUFVFNTDAWYGP5HFEVNVJN6Q4CQN2BB3A, seg_pos={0 0}, upload=476992]: Uploading piece... Dec 28 07:24:32.117199886 (543h55m31.988282931s) [noise=true , address=145.79.227.14:4839 , piece_id=HWBXXJKYTV7C6TOBWMK7U5QJVRROMVPLZLPAP7URZ2E6J6TKBOVA, seg_pos={0 0}, upload=476992]: Uploading piece... Dec 28 07:24:32.118394961 (543h55m31.989477996s) [noise=true , address=145.79.227.37:4567 , piece_id=OPWBMOJSFIVAV2ZQVTSPIUU3RWB6R35COP75KWQW7N4EM3EUE3MA, seg_pos={0 0}, upload=476992]: Uploading piece... Dec 28 07:24:32.119110775 (543h55m31.99019382s) [noise=true , address=77.90.31.13:4626 , piece_id=7DPB5UBH7ABVQEHEZAJRVY5LTLSFU3D5DIZH2NZLYAKUEHZ4UHDA, seg_pos={0 0}, upload=476992]: Uploading piece... Dec 28 07:24:32.122486352 (543h55m31.993569397s) [noise=true , address=45.91.116.13:4156 , piece_id=R7RVEBAT5VGQZHUHRLEPBS7STP7XVA6AY4YSLPRIXJKUVKCFF4VQ, seg_pos={0 0}, upload=476992]: Uploading piece... Dec 28 07:24:32.124638781 (543h55m31.995721816s) [noise=true , address=145.79.227.16:4911 , piece_id=HDA2REQUIC2Z7XRKP77UDNOWFVITNSXSMNFQA7NHIVYJNBB6SIZQ, seg_pos={0 0}, upload=476992]: Uploading piece... Dec 28 07:24:32.124849412 (543h55m31.995932457s) [noise=true , address=145.79.227.18:4007 , piece_id=TZ6KLDXOBKVHULQB3E4WLXRACWPTA6C5JZFWITYFFSXK2D5KOCXA, seg_pos={0 0}, upload=476992]: Uploading piece... Dec 28 07:24:32.126406407 (543h55m31.997489442s) [noise=true , address=45.91.116.10:4084 , piece_id=HVGRMOQXOHAFUNUXY2J42H5OJL6XDEMLO3BLROQEFJ6INUAK4I4A, seg_pos={0 0}, upload=476992]: Uploading piece... Dec 28 07:24:32.127679831 (543h55m31.998762886s) [noise=true , address=145.79.227.42:4123 , piece_id=TIHM4TDHHEXOITN4UL6ZTQUGPD3KU67QNH5Y2HZX7IYQ6EBHKXUQ, seg_pos={0 0}, upload=476992]: Uploading piece... Dec 28 07:24:33.749682045 (543h55m33.62076508s) [upload=476992]: Next returned nil segment Dec 28 07:24:33.749699568 (543h55m33.620782613s) [upload=476992]: Waiting for error group managing segments... Dec 28 07:24:34.119458798 (543h55m33.990541843s) [noise=true , address=145.79.227.45:4231 , piece_id=M4CCTNOTHYYE2YXIGKONI26OH7W2I4R3IQJH2TXY6ZFKLRYTBPNA, seg_pos={0 0}, upload=476992]: Done uploading piece. err: <nil> Dec 28 07:24:34.121601728 (543h55m33.992684763s) [noise=true , address=145.79.227.17:4972 , piece_id=2DEF4NOZF5IEFEHLS3C245RYLN6TKKHIEYAVXJWK2EZDTDB5LZ6A, seg_pos={0 0}, upload=476992]: Done uploading piece. err: <nil> Dec 28 07:24:34.128325031 (543h55m33.999408066s) [noise=true , address=145.79.227.37:4567 , piece_id=OPWBMOJSFIVAV2ZQVTSPIUU3RWB6R35COP75KWQW7N4EM3EUE3MA, seg_pos={0 0}, upload=476992]: Done uploading piece. err: <nil> Dec 28 07:24:34.128912158 (543h55m33.999995203s) [noise=true , address=145.79.227.16:4911 , piece_id=HDA2REQUIC2Z7XRKP77UDNOWFVITNSXSMNFQA7NHIVYJNBB6SIZQ, seg_pos={0 0}, upload=476992]: Done uploading piece. err: <nil> Dec 28 07:24:34.131137215 (543h55m34.00222025s) [noise=true , address=145.79.227.42:4123 , piece_id=TIHM4TDHHEXOITN4UL6ZTQUGPD3KU67QNH5Y2HZX7IYQ6EBHKXUQ, seg_pos={0 0}, upload=476992]: Done uploading piece. err: <nil> Dec 28 07:24:34.133603162 (543h55m34.004686197s) [noise=true , address=145.79.227.7:4293 , piece_id=ILHS7U65IXGQUHGUZOSKDLR75D2ADT6VMY2DF3Q44JGA6NC35YQA, seg_pos={0 0}, upload=476992]: Done uploading piece. err: <nil> Dec 28 07:24:34.140904865 (543h55m34.0119879s) [noise=true , address=145.79.227.35:4481 , piece_id=45CJ7I6U6OJLKJNNCECWBTRWKL3MDXPAP5TSWR57JIXAQ7QTF5CQ, seg_pos={0 0}, upload=476992]: Done uploading piece. err: <nil> Dec 28 07:24:34.146817494 (543h55m34.017900539s) [noise=true , address=145.79.227.14:4839 , piece_id=HWBXXJKYTV7C6TOBWMK7U5QJVRROMVPLZLPAP7URZ2E6J6TKBOVA, seg_pos={0 0}, upload=476992]: Done uploading piece. err: <nil> Dec 28 07:24:34.146901404 (543h55m34.017984449s) [noise=true , address=145.79.227.31:4798 , piece_id=6MA7SUPGRYZ7RZSIURS3IFEE4B7VRDFWLYVTWCQFHSYMGX5ULP6A, seg_pos={0 0}, upload=476992]: Done uploading piece. err: <nil> Dec 28 07:24:34.154751622 (543h55m34.025834657s) [noise=true , address=145.79.227.18:4007 , piece_id=TZ6KLDXOBKVHULQB3E4WLXRACWPTA6C5JZFWITYFFSXK2D5KOCXA, seg_pos={0 0}, upload=476992]: Done uploading piece. err: <nil> Dec 28 07:24:34.362215111 (543h55m34.233298146s) [noise=true , address=45.91.116.12:4225 , piece_id=FEEGNTEUKADXJA6Q525AUFVFNTDAWYGP5HFEVNVJN6Q4CQN2BB3A, seg_pos={0 0}, upload=476992]: Done uploading piece. err: <nil> Dec 28 07:24:34.372017326 (543h55m34.243100371s) [noise=true , address=45.91.116.13:4156 , piece_id=R7RVEBAT5VGQZHUHRLEPBS7STP7XVA6AY4YSLPRIXJKUVKCFF4VQ, seg_pos={0 0}, upload=476992]: Done uploading piece. err: <nil> Dec 28 07:24:34.565728377 (543h55m34.436811422s) [noise=true , address=45.91.116.10:4084 , piece_id=HVGRMOQXOHAFUNUXY2J42H5OJL6XDEMLO3BLROQEFJ6INUAK4I4A, seg_pos={0 0}, upload=476992]: Done uploading piece. err: <nil> Dec 28 07:24:34.565747725 (543h55m34.43683076s) [seg_pos={0 0}, upload=476992]: Segment reached optimal threshold of 13 pieces. Dec 28 07:24:34.566162044 (543h55m34.437245089s) [noise=true , address=77.90.31.13:4626 , piece_id=7DPB5UBH7ABVQEHEZAJRVY5LTLSFU3D5DIZH2NZLYAKUEHZ4UHDA, seg_pos={0 0}, upload=476992]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu): protocol: expected piece hash; context canceled; EOF Dec 28 07:24:34.566169999 (543h55m34.437253044s) [noise=true , address=77.90.31.13:4626 , piece_id=7DPB5UBH7ABVQEHEZAJRVY5LTLSFU3D5DIZH2NZLYAKUEHZ4UHDA, seg_pos={0 0}, upload=476992]: Long tail cancellation due to optimal threshold Dec 28 07:24:34.566514885 (543h55m34.437597921s) [noise=true , address=45.91.116.5:4297 , piece_id=TRXYL52PWBQI3U3H5S7XFNEVTIKZXJXVUL7B5H3YL7RWQNVAWSDA, seg_pos={0 0}, upload=476992]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: write tcp 188.166.225.170:41944->45.91.116.5:4297: use of closed network connection; write tcp 188.166.225.170:41944->45.91.116.5:4297: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41944->45.91.116.5:4297: use of closed network connection Dec 28 07:24:34.566522660 (543h55m34.437605695s) [noise=true , address=45.91.116.5:4297 , piece_id=TRXYL52PWBQI3U3H5S7XFNEVTIKZXJXVUL7B5H3YL7RWQNVAWSDA, seg_pos={0 0}, upload=476992]: Long tail cancellation due to optimal threshold Dec 28 07:24:34.566883597 (543h55m34.437966632s) [noise=true , address=45.91.116.8:4046 , piece_id=2FECWK73M4LDD3WHQTQY53KETVJY4FHMSDABWBLOLODMZIJZVNFQ, seg_pos={0 0}, upload=476992]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H): protocol: context canceled; context canceled Dec 28 07:24:34.566890451 (543h55m34.437973486s) [noise=true , address=45.91.116.8:4046 , piece_id=2FECWK73M4LDD3WHQTQY53KETVJY4FHMSDABWBLOLODMZIJZVNFQ, seg_pos={0 0}, upload=476992]: Long tail cancellation due to optimal threshold Dec 28 07:24:34.566907002 (543h55m34.437990037s) [seg_pos={0 0}, upload=476992]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 07:24:34.567001582 (543h55m34.438084617s) [upload=476992]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 07:24:34.567006662 (543h55m34.438089697s) [upload=476992]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 07:24:34.862454529 (543h55m34.733537564s) [upload=476992]: Done uploading segments... Dec 28 07:24:34.862480950 (543h55m34.733563985s) [upload=476992]: Upload finished. err: <nil> Duration: 6.016676976s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 24 out of 60 is failed (threshold=6) [......X.X...XX......X.X..X..XX.XXXXXXX.X...X..X.XX.....X.X.X] duration: 5.001368744s trace: https://jaeger.dev.storj.io/trace/26ee7c46446453d4 FAIL (2025-12-28 07:25:31 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.00139246s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 24 out of 60 is failed (threshold=3) [X.XXX.XX.....X...XXX.XX.............XX.X...X...X.XXX..X.XX.X] duration: 30.002965639s trace: https://jaeger.dev.storj.io/trace/19034703e3293256 FAIL (2025-12-28 07:25:01 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 28 07:24:31.869041095 (543h55m31.740124131s) [upload=476992]: Starting upload Dec 28 07:24:31.869091793 (543h55m31.740174838s) [upload=476992]: Done starting upload Dec 28 07:24:31.869203846 (543h55m31.740286901s) [upload=476992]: Uploading segments... Dec 28 07:24:31.869341798 (543h55m31.740424854s) [upload=476992]: Got next segment. Inline: false Dec 28 07:24:31.869387505 (543h55m31.74047054s) [upload=476992]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 07:24:31.869392474 (543h55m31.74047551s) [upload=476992]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 07:24:32.112124481 (543h55m31.983207526s) [seg_pos={0 0}, upload=476992]: Begin upload segment... Dec 28 07:24:32.112187041 (543h55m31.983270086s) [seg_pos={0 0}, upload=476992]: Done begin upload segment. Dec 28 07:24:32.112262514 (543h55m31.983345549s) [noise=true , address=145.79.227.45:4231 , piece_id=M4CCTNOTHYYE2YXIGKONI26OH7W2I4R3IQJH2TXY6ZFKLRYTBPNA, seg_pos={0 0}, upload=476992]: Uploading piece... Dec 28 07:24:32.112347206 (543h55m31.983430251s) [noise=true , address=145.79.227.31:4798 , piece_id=6MA7SUPGRYZ7RZSIURS3IFEE4B7VRDFWLYVTWCQFHSYMGX5ULP6A, seg_pos={0 0}, upload=476992]: Uploading piece... Dec 28 07:24:32.112868147 (543h55m31.983951192s) [noise=true , address=145.79.227.7:4293 , piece_id=ILHS7U65IXGQUHGUZOSKDLR75D2ADT6VMY2DF3Q44JGA6NC35YQA, seg_pos={0 0}, upload=476992]: Uploading piece... Dec 28 07:24:32.113275393 (543h55m31.984358438s) [noise=true , address=145.79.227.17:4972 , piece_id=2DEF4NOZF5IEFEHLS3C245RYLN6TKKHIEYAVXJWK2EZDTDB5LZ6A, seg_pos={0 0}, upload=476992]: Uploading piece... Dec 28 07:24:32.114689235 (543h55m31.98577228s) [noise=true , address=45.91.116.5:4297 , piece_id=TRXYL52PWBQI3U3H5S7XFNEVTIKZXJXVUL7B5H3YL7RWQNVAWSDA, seg_pos={0 0}, upload=476992]: Uploading piece... Dec 28 07:24:32.115243370 (543h55m31.986326416s) [noise=true , address=145.79.227.35:4481 , piece_id=45CJ7I6U6OJLKJNNCECWBTRWKL3MDXPAP5TSWR57JIXAQ7QTF5CQ, seg_pos={0 0}, upload=476992]: Uploading piece... Dec 28 07:24:32.115709027 (543h55m31.986792072s) [noise=true , address=45.91.116.8:4046 , piece_id=2FECWK73M4LDD3WHQTQY53KETVJY4FHMSDABWBLOLODMZIJZVNFQ, seg_pos={0 0}, upload=476992]: Uploading piece... Dec 28 07:24:32.116664757 (543h55m31.987747792s) [noise=true , address=45.91.116.12:4225 , piece_id=FEEGNTEUKADXJA6Q525AUFVFNTDAWYGP5HFEVNVJN6Q4CQN2BB3A, seg_pos={0 0}, upload=476992]: Uploading piece... Dec 28 07:24:32.117199886 (543h55m31.988282931s) [noise=true , address=145.79.227.14:4839 , piece_id=HWBXXJKYTV7C6TOBWMK7U5QJVRROMVPLZLPAP7URZ2E6J6TKBOVA, seg_pos={0 0}, upload=476992]: Uploading piece... Dec 28 07:24:32.118394961 (543h55m31.989477996s) [noise=true , address=145.79.227.37:4567 , piece_id=OPWBMOJSFIVAV2ZQVTSPIUU3RWB6R35COP75KWQW7N4EM3EUE3MA, seg_pos={0 0}, upload=476992]: Uploading piece... Dec 28 07:24:32.119110775 (543h55m31.99019382s) [noise=true , address=77.90.31.13:4626 , piece_id=7DPB5UBH7ABVQEHEZAJRVY5LTLSFU3D5DIZH2NZLYAKUEHZ4UHDA, seg_pos={0 0}, upload=476992]: Uploading piece... Dec 28 07:24:32.122486352 (543h55m31.993569397s) [noise=true , address=45.91.116.13:4156 , piece_id=R7RVEBAT5VGQZHUHRLEPBS7STP7XVA6AY4YSLPRIXJKUVKCFF4VQ, seg_pos={0 0}, upload=476992]: Uploading piece... Dec 28 07:24:32.124638781 (543h55m31.995721816s) [noise=true , address=145.79.227.16:4911 , piece_id=HDA2REQUIC2Z7XRKP77UDNOWFVITNSXSMNFQA7NHIVYJNBB6SIZQ, seg_pos={0 0}, upload=476992]: Uploading piece... Dec 28 07:24:32.124849412 (543h55m31.995932457s) [noise=true , address=145.79.227.18:4007 , piece_id=TZ6KLDXOBKVHULQB3E4WLXRACWPTA6C5JZFWITYFFSXK2D5KOCXA, seg_pos={0 0}, upload=476992]: Uploading piece... Dec 28 07:24:32.126406407 (543h55m31.997489442s) [noise=true , address=45.91.116.10:4084 , piece_id=HVGRMOQXOHAFUNUXY2J42H5OJL6XDEMLO3BLROQEFJ6INUAK4I4A, seg_pos={0 0}, upload=476992]: Uploading piece... Dec 28 07:24:32.127679831 (543h55m31.998762886s) [noise=true , address=145.79.227.42:4123 , piece_id=TIHM4TDHHEXOITN4UL6ZTQUGPD3KU67QNH5Y2HZX7IYQ6EBHKXUQ, seg_pos={0 0}, upload=476992]: Uploading piece... Dec 28 07:24:33.749682045 (543h55m33.62076508s) [upload=476992]: Next returned nil segment Dec 28 07:24:33.749699568 (543h55m33.620782613s) [upload=476992]: Waiting for error group managing segments... Dec 28 07:24:34.119458798 (543h55m33.990541843s) [noise=true , address=145.79.227.45:4231 , piece_id=M4CCTNOTHYYE2YXIGKONI26OH7W2I4R3IQJH2TXY6ZFKLRYTBPNA, seg_pos={0 0}, upload=476992]: Done uploading piece. err: <nil> Dec 28 07:24:34.121601728 (543h55m33.992684763s) [noise=true , address=145.79.227.17:4972 , piece_id=2DEF4NOZF5IEFEHLS3C245RYLN6TKKHIEYAVXJWK2EZDTDB5LZ6A, seg_pos={0 0}, upload=476992]: Done uploading piece. err: <nil> Dec 28 07:24:34.128325031 (543h55m33.999408066s) [noise=true , address=145.79.227.37:4567 , piece_id=OPWBMOJSFIVAV2ZQVTSPIUU3RWB6R35COP75KWQW7N4EM3EUE3MA, seg_pos={0 0}, upload=476992]: Done uploading piece. err: <nil> Dec 28 07:24:34.128912158 (543h55m33.999995203s) [noise=true , address=145.79.227.16:4911 , piece_id=HDA2REQUIC2Z7XRKP77UDNOWFVITNSXSMNFQA7NHIVYJNBB6SIZQ, seg_pos={0 0}, upload=476992]: Done uploading piece. err: <nil> Dec 28 07:24:34.131137215 (543h55m34.00222025s) [noise=true , address=145.79.227.42:4123 , piece_id=TIHM4TDHHEXOITN4UL6ZTQUGPD3KU67QNH5Y2HZX7IYQ6EBHKXUQ, seg_pos={0 0}, upload=476992]: Done uploading piece. err: <nil> Dec 28 07:24:34.133603162 (543h55m34.004686197s) [noise=true , address=145.79.227.7:4293 , piece_id=ILHS7U65IXGQUHGUZOSKDLR75D2ADT6VMY2DF3Q44JGA6NC35YQA, seg_pos={0 0}, upload=476992]: Done uploading piece. err: <nil> Dec 28 07:24:34.140904865 (543h55m34.0119879s) [noise=true , address=145.79.227.35:4481 , piece_id=45CJ7I6U6OJLKJNNCECWBTRWKL3MDXPAP5TSWR57JIXAQ7QTF5CQ, seg_pos={0 0}, upload=476992]: Done uploading piece. err: <nil> Dec 28 07:24:34.146817494 (543h55m34.017900539s) [noise=true , address=145.79.227.14:4839 , piece_id=HWBXXJKYTV7C6TOBWMK7U5QJVRROMVPLZLPAP7URZ2E6J6TKBOVA, seg_pos={0 0}, upload=476992]: Done uploading piece. err: <nil> Dec 28 07:24:34.146901404 (543h55m34.017984449s) [noise=true , address=145.79.227.31:4798 , piece_id=6MA7SUPGRYZ7RZSIURS3IFEE4B7VRDFWLYVTWCQFHSYMGX5ULP6A, seg_pos={0 0}, upload=476992]: Done uploading piece. err: <nil> Dec 28 07:24:34.154751622 (543h55m34.025834657s) [noise=true , address=145.79.227.18:4007 , piece_id=TZ6KLDXOBKVHULQB3E4WLXRACWPTA6C5JZFWITYFFSXK2D5KOCXA, seg_pos={0 0}, upload=476992]: Done uploading piece. err: <nil> Dec 28 07:24:34.362215111 (543h55m34.233298146s) [noise=true , address=45.91.116.12:4225 , piece_id=FEEGNTEUKADXJA6Q525AUFVFNTDAWYGP5HFEVNVJN6Q4CQN2BB3A, seg_pos={0 0}, upload=476992]: Done uploading piece. err: <nil> Dec 28 07:24:34.372017326 (543h55m34.243100371s) [noise=true , address=45.91.116.13:4156 , piece_id=R7RVEBAT5VGQZHUHRLEPBS7STP7XVA6AY4YSLPRIXJKUVKCFF4VQ, seg_pos={0 0}, upload=476992]: Done uploading piece. err: <nil> Dec 28 07:24:34.565728377 (543h55m34.436811422s) [noise=true , address=45.91.116.10:4084 , piece_id=HVGRMOQXOHAFUNUXY2J42H5OJL6XDEMLO3BLROQEFJ6INUAK4I4A, seg_pos={0 0}, upload=476992]: Done uploading piece. err: <nil> Dec 28 07:24:34.565747725 (543h55m34.43683076s) [seg_pos={0 0}, upload=476992]: Segment reached optimal threshold of 13 pieces. Dec 28 07:24:34.566162044 (543h55m34.437245089s) [noise=true , address=77.90.31.13:4626 , piece_id=7DPB5UBH7ABVQEHEZAJRVY5LTLSFU3D5DIZH2NZLYAKUEHZ4UHDA, seg_pos={0 0}, upload=476992]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu): protocol: expected piece hash; context canceled; EOF Dec 28 07:24:34.566169999 (543h55m34.437253044s) [noise=true , address=77.90.31.13:4626 , piece_id=7DPB5UBH7ABVQEHEZAJRVY5LTLSFU3D5DIZH2NZLYAKUEHZ4UHDA, seg_pos={0 0}, upload=476992]: Long tail cancellation due to optimal threshold Dec 28 07:24:34.566514885 (543h55m34.437597921s) [noise=true , address=45.91.116.5:4297 , piece_id=TRXYL52PWBQI3U3H5S7XFNEVTIKZXJXVUL7B5H3YL7RWQNVAWSDA, seg_pos={0 0}, upload=476992]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: write tcp 188.166.225.170:41944->45.91.116.5:4297: use of closed network connection; write tcp 188.166.225.170:41944->45.91.116.5:4297: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41944->45.91.116.5:4297: use of closed network connection Dec 28 07:24:34.566522660 (543h55m34.437605695s) [noise=true , address=45.91.116.5:4297 , piece_id=TRXYL52PWBQI3U3H5S7XFNEVTIKZXJXVUL7B5H3YL7RWQNVAWSDA, seg_pos={0 0}, upload=476992]: Long tail cancellation due to optimal threshold Dec 28 07:24:34.566883597 (543h55m34.437966632s) [noise=true , address=45.91.116.8:4046 , piece_id=2FECWK73M4LDD3WHQTQY53KETVJY4FHMSDABWBLOLODMZIJZVNFQ, seg_pos={0 0}, upload=476992]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H): protocol: context canceled; context canceled Dec 28 07:24:34.566890451 (543h55m34.437973486s) [noise=true , address=45.91.116.8:4046 , piece_id=2FECWK73M4LDD3WHQTQY53KETVJY4FHMSDABWBLOLODMZIJZVNFQ, seg_pos={0 0}, upload=476992]: Long tail cancellation due to optimal threshold Dec 28 07:24:34.566907002 (543h55m34.437990037s) [seg_pos={0 0}, upload=476992]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 07:24:34.567001582 (543h55m34.438084617s) [upload=476992]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 07:24:34.567006662 (543h55m34.438089697s) [upload=476992]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 07:24:34.862454529 (543h55m34.733537564s) [upload=476992]: Done uploading segments... Dec 28 07:24:34.862480950 (543h55m34.733563985s) [upload=476992]: Upload finished. err: <nil> Duration: 30.002993743s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 23 out of 60 is failed (threshold=6) [.......X.X...XX......X.X..X..XX.XXXXXXX.X...X..X.XX.....X.X.] duration: 4.937316945s trace: https://jaeger.dev.storj.io/trace/2dd4887ac99c1269 FAIL (2025-12-28 07:24:38 UTC) orbiter warning: context deadline exceeded Duration: 4.937336464s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 23 out of 60 is failed (threshold=3) [.X.XXX.XX.....X...XXX.XX.............XX.X...X...X.XXX..X.XX.] duration: 6.885526378s trace: https://jaeger.dev.storj.io/trace/3e6f942556e7a9c3 FAIL (2025-12-28 07:23:49 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 07:22:45.610743896 (543h53m45.481826941s) [upload=476966]: Starting upload Dec 28 07:22:45.610837574 (543h53m45.481920609s) [upload=476966]: Done starting upload Dec 28 07:22:45.611014792 (543h53m45.482097837s) [upload=476966]: Uploading segments... Dec 28 07:22:45.611057062 (543h53m45.482140117s) [upload=476966]: Got next segment. Inline: false Dec 28 07:22:45.611086568 (543h53m45.482169603s) [upload=476966]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 07:22:45.611091037 (543h53m45.482174072s) [upload=476966]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 07:22:45.858215237 (543h53m45.729298272s) [seg_pos={0 0}, upload=476966]: Begin upload segment... Dec 28 07:22:45.858276013 (543h53m45.729359058s) [seg_pos={0 0}, upload=476966]: Done begin upload segment. Dec 28 07:22:45.858320016 (543h53m45.729403061s) [noise=true , address=145.79.227.37:4565 , piece_id=U2I2Y5NH2W3YBESB4TAU32MGCQXFSTU4QGCEEV6NSEAOIG5XW4GA, seg_pos={0 0}, upload=476966]: Uploading piece... Dec 28 07:22:45.858363088 (543h53m45.729446123s) [noise=true , address=145.79.227.51:4410 , piece_id=JQASZZJWZDR2JQT2OJXDHITPLQXCIEXMJGSCMX5DTPVX3RQMM2NQ, seg_pos={0 0}, upload=476966]: Uploading piece... Dec 28 07:22:45.858847751 (543h53m45.729930786s) [noise=true , address=145.79.227.14:4840 , piece_id=OQCEKFYA5LCTM54QK7N2AU7REKH7LLIDTQQA36ZRGETNE5LH7RVQ, seg_pos={0 0}, upload=476966]: Uploading piece... Dec 28 07:22:45.859174212 (543h53m45.730257247s) [noise=true , address=145.79.227.48:4303 , piece_id=7GOKPAY34APNO4EA5PWGQ3SAM3M3GUPJ26WNX7C4JOFPU4DKYZTQ, seg_pos={0 0}, upload=476966]: Uploading piece... Dec 28 07:22:45.859638537 (543h53m45.730721582s) [noise=true , address=45.91.116.11:4118 , piece_id=FZJBERFQXEM3U3BBXZHOVAAIQ4JGF5WQNFRYJQPGIWZUPTHRUNOA, seg_pos={0 0}, upload=476966]: Uploading piece... Dec 28 07:22:45.859645159 (543h53m45.730728204s) [noise=true , address=77.90.31.21:4877 , piece_id=VL7SFEMP2CWTNORKPSIY7ZUU67RL3FJPMO2GE32ENDTSY3QORZ3A, seg_pos={0 0}, upload=476966]: Uploading piece... Dec 28 07:22:45.860028099 (543h53m45.731111144s) [noise=true , address=45.91.116.9:4264 , piece_id=EHL4W2YEH4BCK75UCOX2HC5RF6DLHH34QSGAIKN3MDWCBF5GIAOA, seg_pos={0 0}, upload=476966]: Uploading piece... Dec 28 07:22:45.860140603 (543h53m45.731223648s) [noise=true , address=145.79.227.13:4804 , piece_id=HIWPKDS2YXSH4RC4MNNXR5UK3MYOIUE6BA5WCRCDKRNSHJ5PIJZA, seg_pos={0 0}, upload=476966]: Uploading piece... Dec 28 07:22:45.860329763 (543h53m45.731412808s) [noise=true , address=145.79.227.5:4223 , piece_id=GGJWAJMDUAPYGUAJKC3E423F3EKOEUYG4FDS3HYEDYAI6Y357NAQ, seg_pos={0 0}, upload=476966]: Uploading piece... Dec 28 07:22:45.860632891 (543h53m45.731715926s) [noise=true , address=145.79.227.4:4186 , piece_id=CBHQZFQ7MZ6OLQULYCVW53KPUQXP7P3FVF2XOLUY5A375IORVJNQ, seg_pos={0 0}, upload=476966]: Uploading piece... Dec 28 07:22:45.860710378 (543h53m45.731793423s) [noise=true , address=145.79.227.31:4797 , piece_id=F77TUQ6Q6RIO3AWF4RFJAND2XVM22VNHQCINXP7DH5MP5FL62OKQ, seg_pos={0 0}, upload=476966]: Uploading piece... Dec 28 07:22:45.861008555 (543h53m45.73209159s) [noise=true , address=145.79.227.41:4085 , piece_id=ZVLHHQKSD64E7OA6XQ3XGKHO756F44FKFVQA4GMUZBXKW57GVOSQ, seg_pos={0 0}, upload=476966]: Uploading piece... Dec 28 07:22:45.861139976 (543h53m45.732223011s) [noise=true , address=145.79.227.36:4518 , piece_id=GOTP3DFI2LLXQ5AHXW2BZ3ZDA33MFCNJADK2O7FM4XTALC3OUR7Q, seg_pos={0 0}, upload=476966]: Uploading piece... Dec 28 07:22:45.861360767 (543h53m45.732443812s) [noise=true , address=77.90.31.10:4519 , piece_id=RTY7VXY4TYNS5NIP3D6V6YD2HWHALU7DTCWNENNTF4WH7KXB6B6A, seg_pos={0 0}, upload=476966]: Uploading piece... Dec 28 07:22:45.861578471 (543h53m45.732661516s) [noise=true , address=145.79.227.16:4912 , piece_id=5N7ZTL4TQQA5CYISJEXUYWF3ILDGZ3GBEQTLJKY3Y5HZVZG7SYNA, seg_pos={0 0}, upload=476966]: Uploading piece... Dec 28 07:22:45.861694041 (543h53m45.732777076s) [noise=true , address=45.91.116.14:4191 , piece_id=3PJ6I362WLFW36WTGPVMTTLFVFOCP7D6ZHEHFFAOPHVGLXLCXSWA, seg_pos={0 0}, upload=476966]: Uploading piece... Dec 28 07:22:47.566230203 (543h53m47.437313238s) [upload=476966]: Next returned nil segment Dec 28 07:22:47.566243598 (543h53m47.437326643s) [upload=476966]: Waiting for error group managing segments... Dec 28 07:22:47.850594235 (543h53m47.72167728s) [noise=true , address=145.79.227.16:4912 , piece_id=5N7ZTL4TQQA5CYISJEXUYWF3ILDGZ3GBEQTLJKY3Y5HZVZG7SYNA, seg_pos={0 0}, upload=476966]: Done uploading piece. err: <nil> Dec 28 07:22:47.853865654 (543h53m47.724948699s) [noise=true , address=145.79.227.4:4186 , piece_id=CBHQZFQ7MZ6OLQULYCVW53KPUQXP7P3FVF2XOLUY5A375IORVJNQ, seg_pos={0 0}, upload=476966]: Done uploading piece. err: <nil> Dec 28 07:22:47.854499240 (543h53m47.725582275s) [noise=true , address=145.79.227.37:4565 , piece_id=U2I2Y5NH2W3YBESB4TAU32MGCQXFSTU4QGCEEV6NSEAOIG5XW4GA, seg_pos={0 0}, upload=476966]: Done uploading piece. err: <nil> Dec 28 07:22:47.867756795 (543h53m47.73883983s) [noise=true , address=145.79.227.5:4223 , piece_id=GGJWAJMDUAPYGUAJKC3E423F3EKOEUYG4FDS3HYEDYAI6Y357NAQ, seg_pos={0 0}, upload=476966]: Done uploading piece. err: <nil> Dec 28 07:22:47.872036995 (543h53m47.74312003s) [noise=true , address=145.79.227.36:4518 , piece_id=GOTP3DFI2LLXQ5AHXW2BZ3ZDA33MFCNJADK2O7FM4XTALC3OUR7Q, seg_pos={0 0}, upload=476966]: Done uploading piece. err: <nil> Dec 28 07:22:47.872612401 (543h53m47.743695437s) [noise=true , address=145.79.227.41:4085 , piece_id=ZVLHHQKSD64E7OA6XQ3XGKHO756F44FKFVQA4GMUZBXKW57GVOSQ, seg_pos={0 0}, upload=476966]: Done uploading piece. err: <nil> Dec 28 07:22:47.873061517 (543h53m47.744144552s) [noise=true , address=145.79.227.14:4840 , piece_id=OQCEKFYA5LCTM54QK7N2AU7REKH7LLIDTQQA36ZRGETNE5LH7RVQ, seg_pos={0 0}, upload=476966]: Done uploading piece. err: <nil> Dec 28 07:22:47.877706060 (543h53m47.748789105s) [noise=true , address=145.79.227.48:4303 , piece_id=7GOKPAY34APNO4EA5PWGQ3SAM3M3GUPJ26WNX7C4JOFPU4DKYZTQ, seg_pos={0 0}, upload=476966]: Done uploading piece. err: <nil> Dec 28 07:22:47.886797352 (543h53m47.757880397s) [noise=true , address=145.79.227.31:4797 , piece_id=F77TUQ6Q6RIO3AWF4RFJAND2XVM22VNHQCINXP7DH5MP5FL62OKQ, seg_pos={0 0}, upload=476966]: Done uploading piece. err: <nil> Dec 28 07:22:47.886981864 (543h53m47.758064899s) [noise=true , address=145.79.227.51:4410 , piece_id=JQASZZJWZDR2JQT2OJXDHITPLQXCIEXMJGSCMX5DTPVX3RQMM2NQ, seg_pos={0 0}, upload=476966]: Done uploading piece. err: <nil> Dec 28 07:22:47.894187745 (543h53m47.76527079s) [noise=true , address=145.79.227.13:4804 , piece_id=HIWPKDS2YXSH4RC4MNNXR5UK3MYOIUE6BA5WCRCDKRNSHJ5PIJZA, seg_pos={0 0}, upload=476966]: Done uploading piece. err: <nil> Dec 28 07:22:48.238208996 (543h53m48.109292031s) [noise=true , address=45.91.116.14:4191 , piece_id=3PJ6I362WLFW36WTGPVMTTLFVFOCP7D6ZHEHFFAOPHVGLXLCXSWA, seg_pos={0 0}, upload=476966]: Done uploading piece. err: <nil> Dec 28 07:22:48.455304657 (543h53m48.326387692s) [noise=true , address=77.90.31.10:4519 , piece_id=RTY7VXY4TYNS5NIP3D6V6YD2HWHALU7DTCWNENNTF4WH7KXB6B6A, seg_pos={0 0}, upload=476966]: Done uploading piece. err: <nil> Dec 28 07:22:48.455332090 (543h53m48.326415125s) [seg_pos={0 0}, upload=476966]: Segment reached optimal threshold of 13 pieces. Dec 28 07:22:48.455606913 (543h53m48.326689988s) [noise=true , address=77.90.31.21:4877 , piece_id=VL7SFEMP2CWTNORKPSIY7ZUU67RL3FJPMO2GE32ENDTSY3QORZ3A, seg_pos={0 0}, upload=476966]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: expected piece hash; context canceled; EOF Dec 28 07:22:48.455618204 (543h53m48.326701239s) [noise=true , address=77.90.31.21:4877 , piece_id=VL7SFEMP2CWTNORKPSIY7ZUU67RL3FJPMO2GE32ENDTSY3QORZ3A, seg_pos={0 0}, upload=476966]: Long tail cancellation due to optimal threshold Dec 28 07:22:48.455677918 (543h53m48.326760973s) [noise=true , address=45.91.116.11:4118 , piece_id=FZJBERFQXEM3U3BBXZHOVAAIQ4JGF5WQNFRYJQPGIWZUPTHRUNOA, seg_pos={0 0}, upload=476966]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p): protocol: write tcp 188.166.225.170:46236->45.91.116.11:4118: use of closed network connection; write tcp 188.166.225.170:46236->45.91.116.11:4118: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46236->45.91.116.11:4118: use of closed network connection Dec 28 07:22:48.455687426 (543h53m48.326770471s) [noise=true , address=45.91.116.11:4118 , piece_id=FZJBERFQXEM3U3BBXZHOVAAIQ4JGF5WQNFRYJQPGIWZUPTHRUNOA, seg_pos={0 0}, upload=476966]: Long tail cancellation due to optimal threshold Dec 28 07:22:48.455912484 (543h53m48.326995529s) [noise=true , address=45.91.116.9:4264 , piece_id=EHL4W2YEH4BCK75UCOX2HC5RF6DLHH34QSGAIKN3MDWCBF5GIAOA, seg_pos={0 0}, upload=476966]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m): protocol: expected piece hash; context canceled; EOF Dec 28 07:22:48.455922644 (543h53m48.327005689s) [noise=true , address=45.91.116.9:4264 , piece_id=EHL4W2YEH4BCK75UCOX2HC5RF6DLHH34QSGAIKN3MDWCBF5GIAOA, seg_pos={0 0}, upload=476966]: Long tail cancellation due to optimal threshold Dec 28 07:22:48.455947381 (543h53m48.327030446s) [seg_pos={0 0}, upload=476966]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 07:22:48.456039657 (543h53m48.327122702s) [upload=476966]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 07:22:48.456047602 (543h53m48.327130647s) [upload=476966]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 07:22:48.754212302 (543h53m48.625295337s) [upload=476966]: Done uploading segments... Dec 28 07:22:48.754231479 (543h53m48.625314514s) [upload=476966]: Upload finished. err: <nil> Duration: 6.885550954s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 23 out of 60 is failed (threshold=6) [........X.X...XX......X.X..X..XX.XXXXXXX.X...X..X.XX.....X.X] duration: 5.001257591s trace: https://jaeger.dev.storj.io/trace/2dd4887ac99c1269 FAIL (2025-12-28 07:23:46 UTC) orbiter warning: context deadline exceeded Duration: 5.00128878s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 23 out of 60 is failed (threshold=3) [..X.XXX.XX.....X...XXX.XX.............XX.X...X...X.XXX..X.XX] duration: 30.003082089s trace: https://jaeger.dev.storj.io/trace/3e6f942556e7a9c3 FAIL (2025-12-28 07:23:14 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 07:22:45.610743896 (543h53m45.481826941s) [upload=476966]: Starting upload Dec 28 07:22:45.610837574 (543h53m45.481920609s) [upload=476966]: Done starting upload Dec 28 07:22:45.611014792 (543h53m45.482097837s) [upload=476966]: Uploading segments... Dec 28 07:22:45.611057062 (543h53m45.482140117s) [upload=476966]: Got next segment. Inline: false Dec 28 07:22:45.611086568 (543h53m45.482169603s) [upload=476966]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 07:22:45.611091037 (543h53m45.482174072s) [upload=476966]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 07:22:45.858215237 (543h53m45.729298272s) [seg_pos={0 0}, upload=476966]: Begin upload segment... Dec 28 07:22:45.858276013 (543h53m45.729359058s) [seg_pos={0 0}, upload=476966]: Done begin upload segment. Dec 28 07:22:45.858320016 (543h53m45.729403061s) [noise=true , address=145.79.227.37:4565 , piece_id=U2I2Y5NH2W3YBESB4TAU32MGCQXFSTU4QGCEEV6NSEAOIG5XW4GA, seg_pos={0 0}, upload=476966]: Uploading piece... Dec 28 07:22:45.858363088 (543h53m45.729446123s) [noise=true , address=145.79.227.51:4410 , piece_id=JQASZZJWZDR2JQT2OJXDHITPLQXCIEXMJGSCMX5DTPVX3RQMM2NQ, seg_pos={0 0}, upload=476966]: Uploading piece... Dec 28 07:22:45.858847751 (543h53m45.729930786s) [noise=true , address=145.79.227.14:4840 , piece_id=OQCEKFYA5LCTM54QK7N2AU7REKH7LLIDTQQA36ZRGETNE5LH7RVQ, seg_pos={0 0}, upload=476966]: Uploading piece... Dec 28 07:22:45.859174212 (543h53m45.730257247s) [noise=true , address=145.79.227.48:4303 , piece_id=7GOKPAY34APNO4EA5PWGQ3SAM3M3GUPJ26WNX7C4JOFPU4DKYZTQ, seg_pos={0 0}, upload=476966]: Uploading piece... Dec 28 07:22:45.859638537 (543h53m45.730721582s) [noise=true , address=45.91.116.11:4118 , piece_id=FZJBERFQXEM3U3BBXZHOVAAIQ4JGF5WQNFRYJQPGIWZUPTHRUNOA, seg_pos={0 0}, upload=476966]: Uploading piece... Dec 28 07:22:45.859645159 (543h53m45.730728204s) [noise=true , address=77.90.31.21:4877 , piece_id=VL7SFEMP2CWTNORKPSIY7ZUU67RL3FJPMO2GE32ENDTSY3QORZ3A, seg_pos={0 0}, upload=476966]: Uploading piece... Dec 28 07:22:45.860028099 (543h53m45.731111144s) [noise=true , address=45.91.116.9:4264 , piece_id=EHL4W2YEH4BCK75UCOX2HC5RF6DLHH34QSGAIKN3MDWCBF5GIAOA, seg_pos={0 0}, upload=476966]: Uploading piece... Dec 28 07:22:45.860140603 (543h53m45.731223648s) [noise=true , address=145.79.227.13:4804 , piece_id=HIWPKDS2YXSH4RC4MNNXR5UK3MYOIUE6BA5WCRCDKRNSHJ5PIJZA, seg_pos={0 0}, upload=476966]: Uploading piece... Dec 28 07:22:45.860329763 (543h53m45.731412808s) [noise=true , address=145.79.227.5:4223 , piece_id=GGJWAJMDUAPYGUAJKC3E423F3EKOEUYG4FDS3HYEDYAI6Y357NAQ, seg_pos={0 0}, upload=476966]: Uploading piece... Dec 28 07:22:45.860632891 (543h53m45.731715926s) [noise=true , address=145.79.227.4:4186 , piece_id=CBHQZFQ7MZ6OLQULYCVW53KPUQXP7P3FVF2XOLUY5A375IORVJNQ, seg_pos={0 0}, upload=476966]: Uploading piece... Dec 28 07:22:45.860710378 (543h53m45.731793423s) [noise=true , address=145.79.227.31:4797 , piece_id=F77TUQ6Q6RIO3AWF4RFJAND2XVM22VNHQCINXP7DH5MP5FL62OKQ, seg_pos={0 0}, upload=476966]: Uploading piece... Dec 28 07:22:45.861008555 (543h53m45.73209159s) [noise=true , address=145.79.227.41:4085 , piece_id=ZVLHHQKSD64E7OA6XQ3XGKHO756F44FKFVQA4GMUZBXKW57GVOSQ, seg_pos={0 0}, upload=476966]: Uploading piece... Dec 28 07:22:45.861139976 (543h53m45.732223011s) [noise=true , address=145.79.227.36:4518 , piece_id=GOTP3DFI2LLXQ5AHXW2BZ3ZDA33MFCNJADK2O7FM4XTALC3OUR7Q, seg_pos={0 0}, upload=476966]: Uploading piece... Dec 28 07:22:45.861360767 (543h53m45.732443812s) [noise=true , address=77.90.31.10:4519 , piece_id=RTY7VXY4TYNS5NIP3D6V6YD2HWHALU7DTCWNENNTF4WH7KXB6B6A, seg_pos={0 0}, upload=476966]: Uploading piece... Dec 28 07:22:45.861578471 (543h53m45.732661516s) [noise=true , address=145.79.227.16:4912 , piece_id=5N7ZTL4TQQA5CYISJEXUYWF3ILDGZ3GBEQTLJKY3Y5HZVZG7SYNA, seg_pos={0 0}, upload=476966]: Uploading piece... Dec 28 07:22:45.861694041 (543h53m45.732777076s) [noise=true , address=45.91.116.14:4191 , piece_id=3PJ6I362WLFW36WTGPVMTTLFVFOCP7D6ZHEHFFAOPHVGLXLCXSWA, seg_pos={0 0}, upload=476966]: Uploading piece... Dec 28 07:22:47.566230203 (543h53m47.437313238s) [upload=476966]: Next returned nil segment Dec 28 07:22:47.566243598 (543h53m47.437326643s) [upload=476966]: Waiting for error group managing segments... Dec 28 07:22:47.850594235 (543h53m47.72167728s) [noise=true , address=145.79.227.16:4912 , piece_id=5N7ZTL4TQQA5CYISJEXUYWF3ILDGZ3GBEQTLJKY3Y5HZVZG7SYNA, seg_pos={0 0}, upload=476966]: Done uploading piece. err: <nil> Dec 28 07:22:47.853865654 (543h53m47.724948699s) [noise=true , address=145.79.227.4:4186 , piece_id=CBHQZFQ7MZ6OLQULYCVW53KPUQXP7P3FVF2XOLUY5A375IORVJNQ, seg_pos={0 0}, upload=476966]: Done uploading piece. err: <nil> Dec 28 07:22:47.854499240 (543h53m47.725582275s) [noise=true , address=145.79.227.37:4565 , piece_id=U2I2Y5NH2W3YBESB4TAU32MGCQXFSTU4QGCEEV6NSEAOIG5XW4GA, seg_pos={0 0}, upload=476966]: Done uploading piece. err: <nil> Dec 28 07:22:47.867756795 (543h53m47.73883983s) [noise=true , address=145.79.227.5:4223 , piece_id=GGJWAJMDUAPYGUAJKC3E423F3EKOEUYG4FDS3HYEDYAI6Y357NAQ, seg_pos={0 0}, upload=476966]: Done uploading piece. err: <nil> Dec 28 07:22:47.872036995 (543h53m47.74312003s) [noise=true , address=145.79.227.36:4518 , piece_id=GOTP3DFI2LLXQ5AHXW2BZ3ZDA33MFCNJADK2O7FM4XTALC3OUR7Q, seg_pos={0 0}, upload=476966]: Done uploading piece. err: <nil> Dec 28 07:22:47.872612401 (543h53m47.743695437s) [noise=true , address=145.79.227.41:4085 , piece_id=ZVLHHQKSD64E7OA6XQ3XGKHO756F44FKFVQA4GMUZBXKW57GVOSQ, seg_pos={0 0}, upload=476966]: Done uploading piece. err: <nil> Dec 28 07:22:47.873061517 (543h53m47.744144552s) [noise=true , address=145.79.227.14:4840 , piece_id=OQCEKFYA5LCTM54QK7N2AU7REKH7LLIDTQQA36ZRGETNE5LH7RVQ, seg_pos={0 0}, upload=476966]: Done uploading piece. err: <nil> Dec 28 07:22:47.877706060 (543h53m47.748789105s) [noise=true , address=145.79.227.48:4303 , piece_id=7GOKPAY34APNO4EA5PWGQ3SAM3M3GUPJ26WNX7C4JOFPU4DKYZTQ, seg_pos={0 0}, upload=476966]: Done uploading piece. err: <nil> Dec 28 07:22:47.886797352 (543h53m47.757880397s) [noise=true , address=145.79.227.31:4797 , piece_id=F77TUQ6Q6RIO3AWF4RFJAND2XVM22VNHQCINXP7DH5MP5FL62OKQ, seg_pos={0 0}, upload=476966]: Done uploading piece. err: <nil> Dec 28 07:22:47.886981864 (543h53m47.758064899s) [noise=true , address=145.79.227.51:4410 , piece_id=JQASZZJWZDR2JQT2OJXDHITPLQXCIEXMJGSCMX5DTPVX3RQMM2NQ, seg_pos={0 0}, upload=476966]: Done uploading piece. err: <nil> Dec 28 07:22:47.894187745 (543h53m47.76527079s) [noise=true , address=145.79.227.13:4804 , piece_id=HIWPKDS2YXSH4RC4MNNXR5UK3MYOIUE6BA5WCRCDKRNSHJ5PIJZA, seg_pos={0 0}, upload=476966]: Done uploading piece. err: <nil> Dec 28 07:22:48.238208996 (543h53m48.109292031s) [noise=true , address=45.91.116.14:4191 , piece_id=3PJ6I362WLFW36WTGPVMTTLFVFOCP7D6ZHEHFFAOPHVGLXLCXSWA, seg_pos={0 0}, upload=476966]: Done uploading piece. err: <nil> Dec 28 07:22:48.455304657 (543h53m48.326387692s) [noise=true , address=77.90.31.10:4519 , piece_id=RTY7VXY4TYNS5NIP3D6V6YD2HWHALU7DTCWNENNTF4WH7KXB6B6A, seg_pos={0 0}, upload=476966]: Done uploading piece. err: <nil> Dec 28 07:22:48.455332090 (543h53m48.326415125s) [seg_pos={0 0}, upload=476966]: Segment reached optimal threshold of 13 pieces. Dec 28 07:22:48.455606913 (543h53m48.326689988s) [noise=true , address=77.90.31.21:4877 , piece_id=VL7SFEMP2CWTNORKPSIY7ZUU67RL3FJPMO2GE32ENDTSY3QORZ3A, seg_pos={0 0}, upload=476966]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: expected piece hash; context canceled; EOF Dec 28 07:22:48.455618204 (543h53m48.326701239s) [noise=true , address=77.90.31.21:4877 , piece_id=VL7SFEMP2CWTNORKPSIY7ZUU67RL3FJPMO2GE32ENDTSY3QORZ3A, seg_pos={0 0}, upload=476966]: Long tail cancellation due to optimal threshold Dec 28 07:22:48.455677918 (543h53m48.326760973s) [noise=true , address=45.91.116.11:4118 , piece_id=FZJBERFQXEM3U3BBXZHOVAAIQ4JGF5WQNFRYJQPGIWZUPTHRUNOA, seg_pos={0 0}, upload=476966]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p): protocol: write tcp 188.166.225.170:46236->45.91.116.11:4118: use of closed network connection; write tcp 188.166.225.170:46236->45.91.116.11:4118: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46236->45.91.116.11:4118: use of closed network connection Dec 28 07:22:48.455687426 (543h53m48.326770471s) [noise=true , address=45.91.116.11:4118 , piece_id=FZJBERFQXEM3U3BBXZHOVAAIQ4JGF5WQNFRYJQPGIWZUPTHRUNOA, seg_pos={0 0}, upload=476966]: Long tail cancellation due to optimal threshold Dec 28 07:22:48.455912484 (543h53m48.326995529s) [noise=true , address=45.91.116.9:4264 , piece_id=EHL4W2YEH4BCK75UCOX2HC5RF6DLHH34QSGAIKN3MDWCBF5GIAOA, seg_pos={0 0}, upload=476966]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m): protocol: expected piece hash; context canceled; EOF Dec 28 07:22:48.455922644 (543h53m48.327005689s) [noise=true , address=45.91.116.9:4264 , piece_id=EHL4W2YEH4BCK75UCOX2HC5RF6DLHH34QSGAIKN3MDWCBF5GIAOA, seg_pos={0 0}, upload=476966]: Long tail cancellation due to optimal threshold Dec 28 07:22:48.455947381 (543h53m48.327030446s) [seg_pos={0 0}, upload=476966]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 07:22:48.456039657 (543h53m48.327122702s) [upload=476966]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 07:22:48.456047602 (543h53m48.327130647s) [upload=476966]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 07:22:48.754212302 (543h53m48.625295337s) [upload=476966]: Done uploading segments... Dec 28 07:22:48.754231479 (543h53m48.625314514s) [upload=476966]: Upload finished. err: <nil> Duration: 30.003103501s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 22 out of 60 is failed (threshold=6) [.........X.X...XX......X.X..X..XX.XXXXXXX.X...X..X.XX.....X.] duration: 4.986465393s trace: https://jaeger.dev.storj.io/trace/1720365573180e2b FAIL (2025-12-28 07:23:01 UTC) orbiter warning: context deadline exceeded Duration: 4.986492837s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [...X.XXX.XX.....X...XXX.XX.............XX.X...X...X.XXX..X.X] duration: 30.003053579s trace: https://jaeger.dev.storj.io/trace/52a46e1d655447ae FAIL (2025-12-28 07:22:41 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, Piece ID: F56MUG6O5S7MO6X4546365DQ2B26ZXUYKA2CCYC5OZ6NDQKUV6HQ) context deadline exceeded; piecestore: (Node ID: 1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, Piece ID: G4L2QFHZPWGMLXC4D72HQ4HQ5OS7VFFCZWU4H4XIVO3NZKZBFJEA) context deadline exceeded ======LOGS====== Dec 28 07:22:12.057482203 (543h53m11.928565248s) [upload=476956]: Starting upload Dec 28 07:22:12.057565813 (543h53m11.928648858s) [upload=476956]: Done starting upload Dec 28 07:22:12.057641096 (543h53m11.928724141s) [upload=476956]: Uploading segments... Dec 28 07:22:12.057838862 (543h53m11.928921907s) [upload=476956]: Got next segment. Inline: false Dec 28 07:22:12.057880913 (543h53m11.928963958s) [upload=476956]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 07:22:12.057887345 (543h53m11.92897039s) [upload=476956]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 07:22:12.308549052 (543h53m12.179632097s) [seg_pos={0 0}, upload=476956]: Begin upload segment... Dec 28 07:22:12.308596322 (543h53m12.179679367s) [seg_pos={0 0}, upload=476956]: Done begin upload segment. Dec 28 07:22:12.308626039 (543h53m12.179709084s) [noise=true , address=145.79.227.19:4079 , piece_id=TZKQFN6HCWJZSBELHATWP5VCHV4OD5XPMXHRDVSVVOESOG6SBOLQ, seg_pos={0 0}, upload=476956]: Uploading piece... Dec 28 07:22:12.308700370 (543h53m12.179783416s) [noise=true , address=145.79.227.16:4910 , piece_id=EEIR67S34IXHIN2XP7BGGTPGQDNNZQPG6EZEJ6RRLH4AB3YRNKJQ, seg_pos={0 0}, upload=476956]: Uploading piece... Dec 28 07:22:12.309544237 (543h53m12.180627282s) [noise=true , address=145.79.227.51:4409 , piece_id=BYIAGVSSFZEKUCBWMU4FSK32SQVRXSTTM3WT42ELDIE3SAV77YYQ, seg_pos={0 0}, upload=476956]: Uploading piece... Dec 28 07:22:12.309886519 (543h53m12.180969564s) [noise=true , address=145.79.227.46:4602 , piece_id=3GW433X33L7SAD7DSDWZLRQQF65TCZP46P23K6ZUORZOW7JOZ7TQ, seg_pos={0 0}, upload=476956]: Uploading piece... Dec 28 07:22:12.310390819 (543h53m12.181473864s) [noise=true , address=145.79.227.11:4732 , piece_id=C4KRRG3VVQIOM5AMLSYNPBLKDTAUGA2M6RL5SWM4YBRXJSQW36UQ, seg_pos={0 0}, upload=476956]: Uploading piece... Dec 28 07:22:12.310825086 (543h53m12.181908131s) [noise=true , address=45.91.116.7:4009 , piece_id=F56MUG6O5S7MO6X4546365DQ2B26ZXUYKA2CCYC5OZ6NDQKUV6HQ, seg_pos={0 0}, upload=476956]: Uploading piece... Dec 28 07:22:12.311230489 (543h53m12.182313534s) [noise=true , address=145.79.227.25:4618 , piece_id=WF72XUJ3NGQHF5KVY2JTJDK3IUBUKVFLRAFVQ7AM5IVSF4RVC6OA, seg_pos={0 0}, upload=476956]: Uploading piece... Dec 28 07:22:12.311635168 (543h53m12.182718204s) [noise=true , address=145.79.227.2:4115 , piece_id=TSRJ53I2SDS655JU7IZ27FGJPD3PVYZKQMCPHR5P2WA5KDHSSKGQ, seg_pos={0 0}, upload=476956]: Uploading piece... Dec 28 07:22:12.312056601 (543h53m12.183139646s) [noise=true , address=77.90.31.13:4627 , piece_id=WVL2JI3P7CGO7Z7A7FIZKLNXER6OFZFEZIGDJFAS6L57MWPQAEOA, seg_pos={0 0}, upload=476956]: Uploading piece... Dec 28 07:22:12.312509914 (543h53m12.183592949s) [noise=true , address=77.90.31.18:4808 , piece_id=427NGXWMULLEETD7EFXR2BJM57A6A2PFCJ4T7SIP3ZCFIJPCXVQA, seg_pos={0 0}, upload=476956]: Uploading piece... Dec 28 07:22:12.312957617 (543h53m12.184040662s) [noise=true , address=145.79.227.7:4293 , piece_id=2T4CQN2AO3WWDFN7ZCLSAOT54NR67AASX4UA5FKVNBLZBSBFYXUQ, seg_pos={0 0}, upload=476956]: Uploading piece... Dec 28 07:22:12.314094893 (543h53m12.185177938s) [noise=true , address=45.91.116.14:4191 , piece_id=XFAZOB3DCQSSZH76TDY2LWELLCK7S7VE6SPJFAVR6TR2UP4RAMDA, seg_pos={0 0}, upload=476956]: Uploading piece... Dec 28 07:22:12.314586849 (543h53m12.185669904s) [noise=true , address=145.79.227.44:4194 , piece_id=MJVYFHEDOTKXJMUVGZ4SBMM5D2LUIF4G4ELGTUL5YY6OL6XGQD3Q, seg_pos={0 0}, upload=476956]: Uploading piece... Dec 28 07:22:12.315059660 (543h53m12.186142705s) [noise=true , address=145.79.227.4:4185 , piece_id=CRR36GFTPDCCY63EGU6UUNJ4RTFUJIYYB422W4GSVCGO5PO2DH2A, seg_pos={0 0}, upload=476956]: Uploading piece... Dec 28 07:22:12.315517932 (543h53m12.186600977s) [noise=true , address=45.91.116.6:4973 , piece_id=3QGG2QIFQCGGSQNCF7YLPUVAXEJX57GV4UP53ZXFO672BVQ7V2ZA, seg_pos={0 0}, upload=476956]: Uploading piece... Dec 28 07:22:12.315955876 (543h53m12.187038921s) [noise=true , address=45.91.116.8:4048 , piece_id=G4L2QFHZPWGMLXC4D72HQ4HQ5OS7VFFCZWU4H4XIVO3NZKZBFJEA, seg_pos={0 0}, upload=476956]: Uploading piece... Dec 28 07:22:14.027790725 (543h53m13.89887377s) [upload=476956]: Next returned nil segment Dec 28 07:22:14.027808440 (543h53m13.898891475s) [upload=476956]: Waiting for error group managing segments... Dec 28 07:22:14.320832882 (543h53m14.191915917s) [noise=true , address=145.79.227.46:4602 , piece_id=3GW433X33L7SAD7DSDWZLRQQF65TCZP46P23K6ZUORZOW7JOZ7TQ, seg_pos={0 0}, upload=476956]: Done uploading piece. err: <nil> Dec 28 07:22:14.335139685 (543h53m14.20622272s) [noise=true , address=145.79.227.7:4293 , piece_id=2T4CQN2AO3WWDFN7ZCLSAOT54NR67AASX4UA5FKVNBLZBSBFYXUQ, seg_pos={0 0}, upload=476956]: Done uploading piece. err: <nil> Dec 28 07:22:14.336457594 (543h53m14.207540639s) [noise=true , address=145.79.227.2:4115 , piece_id=TSRJ53I2SDS655JU7IZ27FGJPD3PVYZKQMCPHR5P2WA5KDHSSKGQ, seg_pos={0 0}, upload=476956]: Done uploading piece. err: <nil> Dec 28 07:22:14.342446598 (543h53m14.213529633s) [noise=true , address=145.79.227.25:4618 , piece_id=WF72XUJ3NGQHF5KVY2JTJDK3IUBUKVFLRAFVQ7AM5IVSF4RVC6OA, seg_pos={0 0}, upload=476956]: Done uploading piece. err: <nil> Dec 28 07:22:14.355521466 (543h53m14.226604501s) [noise=true , address=145.79.227.11:4732 , piece_id=C4KRRG3VVQIOM5AMLSYNPBLKDTAUGA2M6RL5SWM4YBRXJSQW36UQ, seg_pos={0 0}, upload=476956]: Done uploading piece. err: <nil> Dec 28 07:22:14.380282305 (543h53m14.25136534s) [noise=true , address=145.79.227.16:4910 , piece_id=EEIR67S34IXHIN2XP7BGGTPGQDNNZQPG6EZEJ6RRLH4AB3YRNKJQ, seg_pos={0 0}, upload=476956]: Done uploading piece. err: <nil> Dec 28 07:22:14.436435330 (543h53m14.307518375s) [noise=true , address=145.79.227.19:4079 , piece_id=TZKQFN6HCWJZSBELHATWP5VCHV4OD5XPMXHRDVSVVOESOG6SBOLQ, seg_pos={0 0}, upload=476956]: Done uploading piece. err: <nil> Dec 28 07:22:14.450818329 (543h53m14.321901374s) [noise=true , address=145.79.227.51:4409 , piece_id=BYIAGVSSFZEKUCBWMU4FSK32SQVRXSTTM3WT42ELDIE3SAV77YYQ, seg_pos={0 0}, upload=476956]: Done uploading piece. err: <nil> Dec 28 07:22:14.477212487 (543h53m14.348295523s) [noise=true , address=145.79.227.4:4185 , piece_id=CRR36GFTPDCCY63EGU6UUNJ4RTFUJIYYB422W4GSVCGO5PO2DH2A, seg_pos={0 0}, upload=476956]: Done uploading piece. err: <nil> Dec 28 07:22:14.555750159 (543h53m14.426833204s) [noise=true , address=45.91.116.7:4009 , piece_id=F56MUG6O5S7MO6X4546365DQ2B26ZXUYKA2CCYC5OZ6NDQKUV6HQ, seg_pos={0 0}, upload=476956]: Done uploading piece. err: <nil> Dec 28 07:22:14.626294598 (543h53m14.497377643s) [noise=true , address=45.91.116.8:4048 , piece_id=G4L2QFHZPWGMLXC4D72HQ4HQ5OS7VFFCZWU4H4XIVO3NZKZBFJEA, seg_pos={0 0}, upload=476956]: Done uploading piece. err: <nil> Dec 28 07:22:14.674179489 (543h53m14.545262524s) [noise=true , address=45.91.116.6:4973 , piece_id=3QGG2QIFQCGGSQNCF7YLPUVAXEJX57GV4UP53ZXFO672BVQ7V2ZA, seg_pos={0 0}, upload=476956]: Done uploading piece. err: <nil> Dec 28 07:22:14.700339172 (543h53m14.571422218s) [noise=true , address=45.91.116.14:4191 , piece_id=XFAZOB3DCQSSZH76TDY2LWELLCK7S7VE6SPJFAVR6TR2UP4RAMDA, seg_pos={0 0}, upload=476956]: Done uploading piece. err: <nil> Dec 28 07:22:14.700366395 (543h53m14.57144944s) [seg_pos={0 0}, upload=476956]: Segment reached optimal threshold of 13 pieces. Dec 28 07:22:14.700621229 (543h53m14.571704275s) [noise=true , address=145.79.227.44:4194 , piece_id=MJVYFHEDOTKXJMUVGZ4SBMM5D2LUIF4G4ELGTUL5YY6OL6XGQD3Q, seg_pos={0 0}, upload=476956]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh): protocol: expected piece hash; context canceled; EOF Dec 28 07:22:14.700632501 (543h53m14.571715536s) [noise=true , address=145.79.227.44:4194 , piece_id=MJVYFHEDOTKXJMUVGZ4SBMM5D2LUIF4G4ELGTUL5YY6OL6XGQD3Q, seg_pos={0 0}, upload=476956]: Long tail cancellation due to optimal threshold Dec 28 07:22:14.700953212 (543h53m14.572036257s) [noise=true , address=77.90.31.18:4808 , piece_id=427NGXWMULLEETD7EFXR2BJM57A6A2PFCJ4T7SIP3ZCFIJPCXVQA, seg_pos={0 0}, upload=476956]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9): protocol: write tcp 188.166.225.170:40326->77.90.31.18:4808: use of closed network connection; write tcp 188.166.225.170:40326->77.90.31.18:4808: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40326->77.90.31.18:4808: use of closed network connection Dec 28 07:22:14.700970765 (543h53m14.5720538s) [noise=true , address=77.90.31.13:4627 , piece_id=WVL2JI3P7CGO7Z7A7FIZKLNXER6OFZFEZIGDJFAS6L57MWPQAEOA, seg_pos={0 0}, upload=476956]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB): protocol: expected piece hash; context canceled; EOF Dec 28 07:22:14.700983579 (543h53m14.572066614s) [noise=true , address=77.90.31.13:4627 , piece_id=WVL2JI3P7CGO7Z7A7FIZKLNXER6OFZFEZIGDJFAS6L57MWPQAEOA, seg_pos={0 0}, upload=476956]: Long tail cancellation due to optimal threshold Dec 28 07:22:14.701014408 (543h53m14.572097443s) [noise=true , address=77.90.31.18:4808 , piece_id=427NGXWMULLEETD7EFXR2BJM57A6A2PFCJ4T7SIP3ZCFIJPCXVQA, seg_pos={0 0}, upload=476956]: Long tail cancellation due to optimal threshold Dec 28 07:22:14.701035388 (543h53m14.572118433s) [seg_pos={0 0}, upload=476956]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 07:22:14.701135469 (543h53m14.572218514s) [upload=476956]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 07:22:14.701142592 (543h53m14.572225637s) [upload=476956]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 07:22:14.997232552 (543h53m14.868315587s) [upload=476956]: Done uploading segments... Dec 28 07:22:14.997253502 (543h53m14.868336547s) [upload=476956]: Upload finished. err: <nil> Duration: 30.003091171s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 23 out of 60 is failed (threshold=6) [X.........X.X...XX......X.X..X..XX.XXXXXXX.X...X..X.XX.....X] duration: 5.000637989s trace: https://jaeger.dev.storj.io/trace/1720365573180e2b FAIL (2025-12-28 07:22:00 UTC) orbiter warning: context deadline exceeded Duration: 5.000658226s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [....X.XXX.XX.....X...XXX.XX.............XX.X...X...X.XXX..X.] duration: 7.237110391s trace: https://jaeger.dev.storj.io/trace/3b564322d44d26d4 FAIL (2025-12-28 07:21:43 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== Dec 28 07:21:05.115352816 (543h52m4.986435851s) [upload=476939]: Starting upload Dec 28 07:21:05.115439230 (543h52m4.986522275s) [upload=476939]: Done starting upload Dec 28 07:21:05.115508131 (543h52m4.986591166s) [upload=476939]: Uploading segments... Dec 28 07:21:05.115714805 (543h52m4.98679785s) [upload=476939]: Got next segment. Inline: false Dec 28 07:21:05.115760022 (543h52m4.986843067s) [upload=476939]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 07:21:05.115766664 (543h52m4.986849709s) [upload=476939]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 07:21:05.358847276 (543h52m5.229930311s) [seg_pos={0 0}, upload=476939]: Begin upload segment... Dec 28 07:21:05.358892191 (543h52m5.229975226s) [seg_pos={0 0}, upload=476939]: Done begin upload segment. Dec 28 07:21:05.358924943 (543h52m5.230008008s) [noise=true , address=45.91.116.12:4228 , piece_id=G2EYXU3OOAKXZCTSC2PNHEFYWFV44LBTMKCWTOWKHWM3EPGTNKYA, seg_pos={0 0}, upload=476939]: Uploading piece... Dec 28 07:21:05.358967344 (543h52m5.230050389s) [noise=true , address=145.79.227.31:4798 , piece_id=LHDTPWGLEDG4GNSYPCMPDYPFLAAEYYXBEN3LHEQIMN5YOQVNZ35Q, seg_pos={0 0}, upload=476939]: Uploading piece... Dec 28 07:21:05.359794419 (543h52m5.230877464s) [noise=true , address=45.91.116.6:4974 , piece_id=ANBMY5JTMG53V2MC5YXFZQL2E5T5HWEHQNGJEVDG5XZMXJMX27JA, seg_pos={0 0}, upload=476939]: Uploading piece... Dec 28 07:21:05.360064073 (543h52m5.231147118s) [noise=true , address=77.90.31.19:4842 , piece_id=IEH5XUOO3SIRWH4JIDMIF4Q2BZS7LGQQMPOSNZQW63YO77YRED3Q, seg_pos={0 0}, upload=476939]: Uploading piece... Dec 28 07:21:05.360655589 (543h52m5.231738634s) [noise=true , address=145.79.227.15:4873 , piece_id=5S4OA3ENTZYCY5GQZPFPD7IGGXGNQDW3DOKF6NNBVRQQSACJOLDQ, seg_pos={0 0}, upload=476939]: Uploading piece... Dec 28 07:21:05.361125734 (543h52m5.232208779s) [noise=true , address=145.79.227.5:4221 , piece_id=ET3RWI6LU6LIQB264VCRQ3EYTBEM2UC6A2ONOWACSUYNUXKLNXUQ, seg_pos={0 0}, upload=476939]: Uploading piece... Dec 28 07:21:05.361149008 (543h52m5.232232054s) [noise=true , address=145.79.227.37:4565 , piece_id=67R2SYWR2JC7DZC4PAY34RUXPLATVNFJHIYOF34T2K6CQZU5W2UA, seg_pos={0 0}, upload=476939]: Uploading piece... Dec 28 07:21:05.361562997 (543h52m5.232646042s) [noise=true , address=145.79.227.46:4601 , piece_id=S6FJHAH2R6AW2RE4KBKU3SM3RTGPKRP46NC4V7HMJIWXIZLCQVEQ, seg_pos={0 0}, upload=476939]: Uploading piece... Dec 28 07:21:05.361886412 (543h52m5.232969457s) [noise=true , address=145.79.227.2:4115 , piece_id=PDZ4KTF5RD4CPOMVCGR5JE6YA6WTM3TKRMASA3KEOTFPZ2546Z4Q, seg_pos={0 0}, upload=476939]: Uploading piece... Dec 28 07:21:05.362125528 (543h52m5.233208573s) [noise=true , address=145.79.227.38:4977 , piece_id=VGNQMV4VKN2LQLXR6MQPQCT5DAKL26C2KUR3HWODEPYL2FVKMS3A, seg_pos={0 0}, upload=476939]: Uploading piece... Dec 28 07:21:05.362518776 (543h52m5.233601821s) [noise=true , address=77.90.31.14:4664 , piece_id=XTQB4MHYDVZ3F7HDLPZFTN7W7F7XR5DS5AGO2P5SLSQ3XTG2KXZA, seg_pos={0 0}, upload=476939]: Uploading piece... Dec 28 07:21:05.362641310 (543h52m5.233724355s) [noise=true , address=77.90.31.12:4590 , piece_id=UYCGLOKPNB23Y2PSK4K7Z4JXJ3CXVF23UHOOFZPXVFM7OO25ABZQ, seg_pos={0 0}, upload=476939]: Uploading piece... Dec 28 07:21:05.363023809 (543h52m5.234106844s) [noise=true , address=145.79.227.29:4941 , piece_id=2ZZWNSW4MSHFSUVNRXMW5LCT7EJ3Y6A7DVCCERSA27AR6YZ7HKYQ, seg_pos={0 0}, upload=476939]: Uploading piece... Dec 28 07:21:05.363247805 (543h52m5.234331s) [noise=true , address=77.90.31.18:4808 , piece_id=AWOPSSV5LXS6Z7JL5FNGNSOACCTGU3CGOE25Q7EUND4NTF57U76Q, seg_pos={0 0}, upload=476939]: Uploading piece... Dec 28 07:21:05.363679337 (543h52m5.234762372s) [noise=true , address=145.79.227.42:4122 , piece_id=VQ7CPVYYWZOYIMOLKGEMXLBIWAP2QVCBIIF4J4YJSJJHDE5ZVEQQ, seg_pos={0 0}, upload=476939]: Uploading piece... Dec 28 07:21:05.364118092 (543h52m5.235201137s) [noise=true , address=145.79.227.12:4767 , piece_id=HY4GDYUVEEW4MADUIWSQMJTEGTO77J4Z2RYNJ7AAY65GUPTM6SOQ, seg_pos={0 0}, upload=476939]: Uploading piece... Dec 28 07:21:07.082119995 (543h52m6.95320305s) [upload=476939]: Next returned nil segment Dec 28 07:21:07.082151986 (543h52m6.953235021s) [upload=476939]: Waiting for error group managing segments... Dec 28 07:21:07.361016156 (543h52m7.232099201s) [noise=true , address=145.79.227.15:4873 , piece_id=5S4OA3ENTZYCY5GQZPFPD7IGGXGNQDW3DOKF6NNBVRQQSACJOLDQ, seg_pos={0 0}, upload=476939]: Done uploading piece. err: <nil> Dec 28 07:21:07.369638744 (543h52m7.240721789s) [noise=true , address=145.79.227.46:4601 , piece_id=S6FJHAH2R6AW2RE4KBKU3SM3RTGPKRP46NC4V7HMJIWXIZLCQVEQ, seg_pos={0 0}, upload=476939]: Done uploading piece. err: <nil> Dec 28 07:21:07.370379115 (543h52m7.24146215s) [noise=true , address=145.79.227.29:4941 , piece_id=2ZZWNSW4MSHFSUVNRXMW5LCT7EJ3Y6A7DVCCERSA27AR6YZ7HKYQ, seg_pos={0 0}, upload=476939]: Done uploading piece. err: <nil> Dec 28 07:21:07.375733389 (543h52m7.246816434s) [noise=true , address=145.79.227.2:4115 , piece_id=PDZ4KTF5RD4CPOMVCGR5JE6YA6WTM3TKRMASA3KEOTFPZ2546Z4Q, seg_pos={0 0}, upload=476939]: Done uploading piece. err: <nil> Dec 28 07:21:07.378927341 (543h52m7.250010387s) [noise=true , address=145.79.227.42:4122 , piece_id=VQ7CPVYYWZOYIMOLKGEMXLBIWAP2QVCBIIF4J4YJSJJHDE5ZVEQQ, seg_pos={0 0}, upload=476939]: Done uploading piece. err: <nil> Dec 28 07:21:07.384453715 (543h52m7.25553676s) [noise=true , address=145.79.227.5:4221 , piece_id=ET3RWI6LU6LIQB264VCRQ3EYTBEM2UC6A2ONOWACSUYNUXKLNXUQ, seg_pos={0 0}, upload=476939]: Done uploading piece. err: <nil> Dec 28 07:21:07.390662387 (543h52m7.261745432s) [noise=true , address=145.79.227.37:4565 , piece_id=67R2SYWR2JC7DZC4PAY34RUXPLATVNFJHIYOF34T2K6CQZU5W2UA, seg_pos={0 0}, upload=476939]: Done uploading piece. err: <nil> Dec 28 07:21:07.390855194 (543h52m7.26193823s) [noise=true , address=145.79.227.31:4798 , piece_id=LHDTPWGLEDG4GNSYPCMPDYPFLAAEYYXBEN3LHEQIMN5YOQVNZ35Q, seg_pos={0 0}, upload=476939]: Done uploading piece. err: <nil> Dec 28 07:21:07.394285025 (543h52m7.26536807s) [noise=true , address=145.79.227.38:4977 , piece_id=VGNQMV4VKN2LQLXR6MQPQCT5DAKL26C2KUR3HWODEPYL2FVKMS3A, seg_pos={0 0}, upload=476939]: Done uploading piece. err: <nil> Dec 28 07:21:07.397563678 (543h52m7.268646723s) [noise=true , address=145.79.227.12:4767 , piece_id=HY4GDYUVEEW4MADUIWSQMJTEGTO77J4Z2RYNJ7AAY65GUPTM6SOQ, seg_pos={0 0}, upload=476939]: Done uploading piece. err: <nil> Dec 28 07:21:07.595936758 (543h52m7.467019793s) [noise=true , address=45.91.116.12:4228 , piece_id=G2EYXU3OOAKXZCTSC2PNHEFYWFV44LBTMKCWTOWKHWM3EPGTNKYA, seg_pos={0 0}, upload=476939]: Done uploading piece. err: <nil> Dec 28 07:21:07.614567664 (543h52m7.485650699s) [noise=true , address=45.91.116.6:4974 , piece_id=ANBMY5JTMG53V2MC5YXFZQL2E5T5HWEHQNGJEVDG5XZMXJMX27JA, seg_pos={0 0}, upload=476939]: Done uploading piece. err: <nil> Dec 28 07:21:07.626970913 (543h52m7.498053948s) [noise=true , address=77.90.31.12:4590 , piece_id=UYCGLOKPNB23Y2PSK4K7Z4JXJ3CXVF23UHOOFZPXVFM7OO25ABZQ, seg_pos={0 0}, upload=476939]: Done uploading piece. err: <nil> Dec 28 07:21:07.626998746 (543h52m7.498081791s) [seg_pos={0 0}, upload=476939]: Segment reached optimal threshold of 13 pieces. Dec 28 07:21:07.627274681 (543h52m7.498357716s) [noise=true , address=77.90.31.14:4664 , piece_id=XTQB4MHYDVZ3F7HDLPZFTN7W7F7XR5DS5AGO2P5SLSQ3XTG2KXZA, seg_pos={0 0}, upload=476939]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12u52PWsW5g7XTky7sxS5mPFqpMD4AHCXFfera1WZyZF9xf6ma9): protocol: expected piece hash; context canceled; EOF Dec 28 07:21:07.627283268 (543h52m7.498366313s) [noise=true , address=77.90.31.14:4664 , piece_id=XTQB4MHYDVZ3F7HDLPZFTN7W7F7XR5DS5AGO2P5SLSQ3XTG2KXZA, seg_pos={0 0}, upload=476939]: Long tail cancellation due to optimal threshold Dec 28 07:21:07.627561036 (543h52m7.498644072s) [noise=true , address=77.90.31.19:4842 , piece_id=IEH5XUOO3SIRWH4JIDMIF4Q2BZS7LGQQMPOSNZQW63YO77YRED3Q, seg_pos={0 0}, upload=476939]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: expected piece hash; context canceled; EOF Dec 28 07:21:07.627569923 (543h52m7.498652958s) [noise=true , address=77.90.31.19:4842 , piece_id=IEH5XUOO3SIRWH4JIDMIF4Q2BZS7LGQQMPOSNZQW63YO77YRED3Q, seg_pos={0 0}, upload=476939]: Long tail cancellation due to optimal threshold Dec 28 07:21:07.627650426 (543h52m7.498733481s) [noise=true , address=77.90.31.18:4808 , piece_id=AWOPSSV5LXS6Z7JL5FNGNSOACCTGU3CGOE25Q7EUND4NTF57U76Q, seg_pos={0 0}, upload=476939]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9): protocol: expected piece hash; context canceled; EOF Dec 28 07:21:07.627669593 (543h52m7.498752638s) [noise=true , address=77.90.31.18:4808 , piece_id=AWOPSSV5LXS6Z7JL5FNGNSOACCTGU3CGOE25Q7EUND4NTF57U76Q, seg_pos={0 0}, upload=476939]: Long tail cancellation due to optimal threshold Dec 28 07:21:07.627717444 (543h52m7.498800489s) [seg_pos={0 0}, upload=476939]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 07:21:07.627830730 (543h52m7.498913765s) [upload=476939]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 07:21:07.627848454 (543h52m7.498931489s) [upload=476939]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 07:21:07.916479911 (543h52m7.787562946s) [upload=476939]: Done uploading segments... Dec 28 07:21:07.916499609 (543h52m7.787582654s) [upload=476939]: Upload finished. err: <nil> Duration: 7.237140048s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 22 out of 60 is failed (threshold=6) [.X.........X.X...XX......X.X..X..XX.XXXXXXX.X...X..X.XX.....] duration: 5.000921081s trace: https://jaeger.dev.storj.io/trace/584f899091a23e15 FAIL (2025-12-28 07:21:43 UTC) orbiter warning: context deadline exceeded Duration: 5.000944054s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [.....X.XXX.XX.....X...XXX.XX.............XX.X...X...X.XXX..X] duration: 19.58380536s trace: https://jaeger.dev.storj.io/trace/3b564322d44d26d4 FAIL (2025-12-28 07:21:24 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== Dec 28 07:21:05.115352816 (543h52m4.986435851s) [upload=476939]: Starting upload Dec 28 07:21:05.115439230 (543h52m4.986522275s) [upload=476939]: Done starting upload Dec 28 07:21:05.115508131 (543h52m4.986591166s) [upload=476939]: Uploading segments... Dec 28 07:21:05.115714805 (543h52m4.98679785s) [upload=476939]: Got next segment. Inline: false Dec 28 07:21:05.115760022 (543h52m4.986843067s) [upload=476939]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 07:21:05.115766664 (543h52m4.986849709s) [upload=476939]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 07:21:05.358847276 (543h52m5.229930311s) [seg_pos={0 0}, upload=476939]: Begin upload segment... Dec 28 07:21:05.358892191 (543h52m5.229975226s) [seg_pos={0 0}, upload=476939]: Done begin upload segment. Dec 28 07:21:05.358924943 (543h52m5.230008008s) [noise=true , address=45.91.116.12:4228 , piece_id=G2EYXU3OOAKXZCTSC2PNHEFYWFV44LBTMKCWTOWKHWM3EPGTNKYA, seg_pos={0 0}, upload=476939]: Uploading piece... Dec 28 07:21:05.358967344 (543h52m5.230050389s) [noise=true , address=145.79.227.31:4798 , piece_id=LHDTPWGLEDG4GNSYPCMPDYPFLAAEYYXBEN3LHEQIMN5YOQVNZ35Q, seg_pos={0 0}, upload=476939]: Uploading piece... Dec 28 07:21:05.359794419 (543h52m5.230877464s) [noise=true , address=45.91.116.6:4974 , piece_id=ANBMY5JTMG53V2MC5YXFZQL2E5T5HWEHQNGJEVDG5XZMXJMX27JA, seg_pos={0 0}, upload=476939]: Uploading piece... Dec 28 07:21:05.360064073 (543h52m5.231147118s) [noise=true , address=77.90.31.19:4842 , piece_id=IEH5XUOO3SIRWH4JIDMIF4Q2BZS7LGQQMPOSNZQW63YO77YRED3Q, seg_pos={0 0}, upload=476939]: Uploading piece... Dec 28 07:21:05.360655589 (543h52m5.231738634s) [noise=true , address=145.79.227.15:4873 , piece_id=5S4OA3ENTZYCY5GQZPFPD7IGGXGNQDW3DOKF6NNBVRQQSACJOLDQ, seg_pos={0 0}, upload=476939]: Uploading piece... Dec 28 07:21:05.361125734 (543h52m5.232208779s) [noise=true , address=145.79.227.5:4221 , piece_id=ET3RWI6LU6LIQB264VCRQ3EYTBEM2UC6A2ONOWACSUYNUXKLNXUQ, seg_pos={0 0}, upload=476939]: Uploading piece... Dec 28 07:21:05.361149008 (543h52m5.232232054s) [noise=true , address=145.79.227.37:4565 , piece_id=67R2SYWR2JC7DZC4PAY34RUXPLATVNFJHIYOF34T2K6CQZU5W2UA, seg_pos={0 0}, upload=476939]: Uploading piece... Dec 28 07:21:05.361562997 (543h52m5.232646042s) [noise=true , address=145.79.227.46:4601 , piece_id=S6FJHAH2R6AW2RE4KBKU3SM3RTGPKRP46NC4V7HMJIWXIZLCQVEQ, seg_pos={0 0}, upload=476939]: Uploading piece... Dec 28 07:21:05.361886412 (543h52m5.232969457s) [noise=true , address=145.79.227.2:4115 , piece_id=PDZ4KTF5RD4CPOMVCGR5JE6YA6WTM3TKRMASA3KEOTFPZ2546Z4Q, seg_pos={0 0}, upload=476939]: Uploading piece... Dec 28 07:21:05.362125528 (543h52m5.233208573s) [noise=true , address=145.79.227.38:4977 , piece_id=VGNQMV4VKN2LQLXR6MQPQCT5DAKL26C2KUR3HWODEPYL2FVKMS3A, seg_pos={0 0}, upload=476939]: Uploading piece... Dec 28 07:21:05.362518776 (543h52m5.233601821s) [noise=true , address=77.90.31.14:4664 , piece_id=XTQB4MHYDVZ3F7HDLPZFTN7W7F7XR5DS5AGO2P5SLSQ3XTG2KXZA, seg_pos={0 0}, upload=476939]: Uploading piece... Dec 28 07:21:05.362641310 (543h52m5.233724355s) [noise=true , address=77.90.31.12:4590 , piece_id=UYCGLOKPNB23Y2PSK4K7Z4JXJ3CXVF23UHOOFZPXVFM7OO25ABZQ, seg_pos={0 0}, upload=476939]: Uploading piece... Dec 28 07:21:05.363023809 (543h52m5.234106844s) [noise=true , address=145.79.227.29:4941 , piece_id=2ZZWNSW4MSHFSUVNRXMW5LCT7EJ3Y6A7DVCCERSA27AR6YZ7HKYQ, seg_pos={0 0}, upload=476939]: Uploading piece... Dec 28 07:21:05.363247805 (543h52m5.234331s) [noise=true , address=77.90.31.18:4808 , piece_id=AWOPSSV5LXS6Z7JL5FNGNSOACCTGU3CGOE25Q7EUND4NTF57U76Q, seg_pos={0 0}, upload=476939]: Uploading piece... Dec 28 07:21:05.363679337 (543h52m5.234762372s) [noise=true , address=145.79.227.42:4122 , piece_id=VQ7CPVYYWZOYIMOLKGEMXLBIWAP2QVCBIIF4J4YJSJJHDE5ZVEQQ, seg_pos={0 0}, upload=476939]: Uploading piece... Dec 28 07:21:05.364118092 (543h52m5.235201137s) [noise=true , address=145.79.227.12:4767 , piece_id=HY4GDYUVEEW4MADUIWSQMJTEGTO77J4Z2RYNJ7AAY65GUPTM6SOQ, seg_pos={0 0}, upload=476939]: Uploading piece... Dec 28 07:21:07.082119995 (543h52m6.95320305s) [upload=476939]: Next returned nil segment Dec 28 07:21:07.082151986 (543h52m6.953235021s) [upload=476939]: Waiting for error group managing segments... Dec 28 07:21:07.361016156 (543h52m7.232099201s) [noise=true , address=145.79.227.15:4873 , piece_id=5S4OA3ENTZYCY5GQZPFPD7IGGXGNQDW3DOKF6NNBVRQQSACJOLDQ, seg_pos={0 0}, upload=476939]: Done uploading piece. err: <nil> Dec 28 07:21:07.369638744 (543h52m7.240721789s) [noise=true , address=145.79.227.46:4601 , piece_id=S6FJHAH2R6AW2RE4KBKU3SM3RTGPKRP46NC4V7HMJIWXIZLCQVEQ, seg_pos={0 0}, upload=476939]: Done uploading piece. err: <nil> Dec 28 07:21:07.370379115 (543h52m7.24146215s) [noise=true , address=145.79.227.29:4941 , piece_id=2ZZWNSW4MSHFSUVNRXMW5LCT7EJ3Y6A7DVCCERSA27AR6YZ7HKYQ, seg_pos={0 0}, upload=476939]: Done uploading piece. err: <nil> Dec 28 07:21:07.375733389 (543h52m7.246816434s) [noise=true , address=145.79.227.2:4115 , piece_id=PDZ4KTF5RD4CPOMVCGR5JE6YA6WTM3TKRMASA3KEOTFPZ2546Z4Q, seg_pos={0 0}, upload=476939]: Done uploading piece. err: <nil> Dec 28 07:21:07.378927341 (543h52m7.250010387s) [noise=true , address=145.79.227.42:4122 , piece_id=VQ7CPVYYWZOYIMOLKGEMXLBIWAP2QVCBIIF4J4YJSJJHDE5ZVEQQ, seg_pos={0 0}, upload=476939]: Done uploading piece. err: <nil> Dec 28 07:21:07.384453715 (543h52m7.25553676s) [noise=true , address=145.79.227.5:4221 , piece_id=ET3RWI6LU6LIQB264VCRQ3EYTBEM2UC6A2ONOWACSUYNUXKLNXUQ, seg_pos={0 0}, upload=476939]: Done uploading piece. err: <nil> Dec 28 07:21:07.390662387 (543h52m7.261745432s) [noise=true , address=145.79.227.37:4565 , piece_id=67R2SYWR2JC7DZC4PAY34RUXPLATVNFJHIYOF34T2K6CQZU5W2UA, seg_pos={0 0}, upload=476939]: Done uploading piece. err: <nil> Dec 28 07:21:07.390855194 (543h52m7.26193823s) [noise=true , address=145.79.227.31:4798 , piece_id=LHDTPWGLEDG4GNSYPCMPDYPFLAAEYYXBEN3LHEQIMN5YOQVNZ35Q, seg_pos={0 0}, upload=476939]: Done uploading piece. err: <nil> Dec 28 07:21:07.394285025 (543h52m7.26536807s) [noise=true , address=145.79.227.38:4977 , piece_id=VGNQMV4VKN2LQLXR6MQPQCT5DAKL26C2KUR3HWODEPYL2FVKMS3A, seg_pos={0 0}, upload=476939]: Done uploading piece. err: <nil> Dec 28 07:21:07.397563678 (543h52m7.268646723s) [noise=true , address=145.79.227.12:4767 , piece_id=HY4GDYUVEEW4MADUIWSQMJTEGTO77J4Z2RYNJ7AAY65GUPTM6SOQ, seg_pos={0 0}, upload=476939]: Done uploading piece. err: <nil> Dec 28 07:21:07.595936758 (543h52m7.467019793s) [noise=true , address=45.91.116.12:4228 , piece_id=G2EYXU3OOAKXZCTSC2PNHEFYWFV44LBTMKCWTOWKHWM3EPGTNKYA, seg_pos={0 0}, upload=476939]: Done uploading piece. err: <nil> Dec 28 07:21:07.614567664 (543h52m7.485650699s) [noise=true , address=45.91.116.6:4974 , piece_id=ANBMY5JTMG53V2MC5YXFZQL2E5T5HWEHQNGJEVDG5XZMXJMX27JA, seg_pos={0 0}, upload=476939]: Done uploading piece. err: <nil> Dec 28 07:21:07.626970913 (543h52m7.498053948s) [noise=true , address=77.90.31.12:4590 , piece_id=UYCGLOKPNB23Y2PSK4K7Z4JXJ3CXVF23UHOOFZPXVFM7OO25ABZQ, seg_pos={0 0}, upload=476939]: Done uploading piece. err: <nil> Dec 28 07:21:07.626998746 (543h52m7.498081791s) [seg_pos={0 0}, upload=476939]: Segment reached optimal threshold of 13 pieces. Dec 28 07:21:07.627274681 (543h52m7.498357716s) [noise=true , address=77.90.31.14:4664 , piece_id=XTQB4MHYDVZ3F7HDLPZFTN7W7F7XR5DS5AGO2P5SLSQ3XTG2KXZA, seg_pos={0 0}, upload=476939]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12u52PWsW5g7XTky7sxS5mPFqpMD4AHCXFfera1WZyZF9xf6ma9): protocol: expected piece hash; context canceled; EOF Dec 28 07:21:07.627283268 (543h52m7.498366313s) [noise=true , address=77.90.31.14:4664 , piece_id=XTQB4MHYDVZ3F7HDLPZFTN7W7F7XR5DS5AGO2P5SLSQ3XTG2KXZA, seg_pos={0 0}, upload=476939]: Long tail cancellation due to optimal threshold Dec 28 07:21:07.627561036 (543h52m7.498644072s) [noise=true , address=77.90.31.19:4842 , piece_id=IEH5XUOO3SIRWH4JIDMIF4Q2BZS7LGQQMPOSNZQW63YO77YRED3Q, seg_pos={0 0}, upload=476939]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: expected piece hash; context canceled; EOF Dec 28 07:21:07.627569923 (543h52m7.498652958s) [noise=true , address=77.90.31.19:4842 , piece_id=IEH5XUOO3SIRWH4JIDMIF4Q2BZS7LGQQMPOSNZQW63YO77YRED3Q, seg_pos={0 0}, upload=476939]: Long tail cancellation due to optimal threshold Dec 28 07:21:07.627650426 (543h52m7.498733481s) [noise=true , address=77.90.31.18:4808 , piece_id=AWOPSSV5LXS6Z7JL5FNGNSOACCTGU3CGOE25Q7EUND4NTF57U76Q, seg_pos={0 0}, upload=476939]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9): protocol: expected piece hash; context canceled; EOF Dec 28 07:21:07.627669593 (543h52m7.498752638s) [noise=true , address=77.90.31.18:4808 , piece_id=AWOPSSV5LXS6Z7JL5FNGNSOACCTGU3CGOE25Q7EUND4NTF57U76Q, seg_pos={0 0}, upload=476939]: Long tail cancellation due to optimal threshold Dec 28 07:21:07.627717444 (543h52m7.498800489s) [seg_pos={0 0}, upload=476939]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 07:21:07.627830730 (543h52m7.498913765s) [upload=476939]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 07:21:07.627848454 (543h52m7.498931489s) [upload=476939]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 07:21:07.916479911 (543h52m7.787562946s) [upload=476939]: Done uploading segments... Dec 28 07:21:07.916499609 (543h52m7.787582654s) [upload=476939]: Upload finished. err: <nil> Duration: 19.583846947s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 23 out of 60 is failed (threshold=6) [X.X.........X.X...XX......X.X..X..XX.XXXXXXX.X...X..X.XX....] duration: 3.676175239s trace: https://jaeger.dev.storj.io/trace/584f899091a23e15 FAIL (2025-12-28 07:21:00 UTC) orbiter warning: context deadline exceeded Duration: 3.676209545s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 23 out of 60 is failed (threshold=6) [.X.X.........X.X...XX......X.X..X..XX.XXXXXXX.X...X..X.XX...] duration: 4.894136646s trace: https://jaeger.dev.storj.io/trace/584f899091a23e15 FAIL (2025-12-28 07:20:31 UTC) orbiter warning: context deadline exceeded Duration: 4.894166664s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [......X.XXX.XX.....X...XXX.XX.............XX.X...X...X.XXX..] duration: 5.739184649s trace: https://jaeger.dev.storj.io/trace/2fd38217318d19b1 FAIL (2025-12-28 07:19:58 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, Piece ID: DLFRP5UK5FOTW2IMP5PG5Q6J3SQ46BBUEDKJOOSVLOG3TWGUZXQA) context deadline exceeded; piecestore: (Node ID: 12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, Piece ID: ANL43CYDWEHDCQGXMYDNC5IOMA67TUFI7NEJWD2BQPKBBFM4UZRA) context deadline exceeded ======LOGS====== 8 07:18:10.252919992 (543h49m10.124003037s) [upload=476901]: Starting upload Dec 28 07:18:10.253074667 (543h49m10.124157702s) [upload=476901]: Done starting upload Dec 28 07:18:10.253165671 (543h49m10.124248726s) [upload=476901]: Uploading segments... Dec 28 07:18:10.253425986 (543h49m10.124509041s) [upload=476901]: Got next segment. Inline: false Dec 28 07:18:10.253469889 (543h49m10.124552924s) [upload=476901]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 07:18:10.253477404 (543h49m10.124560449s) [upload=476901]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 07:18:10.511603543 (543h49m10.382686598s) [seg_pos={0 0}, upload=476901]: Begin upload segment... Dec 28 07:18:10.511660832 (543h49m10.382743877s) [seg_pos={0 0}, upload=476901]: Done begin upload segment. Dec 28 07:18:10.511676382 (543h49m10.382759417s) [noise=true , address=145.79.227.42:4121 , piece_id=J3FKKSK7ILIFCR5E7OX3OZ7M5VUKESDFU7K2SOWAN6IWEFZYHHHQ, seg_pos={0 0}, upload=476901]: Uploading piece... Dec 28 07:18:10.511707311 (543h49m10.382790416s) [noise=true , address=45.91.116.7:4011 , piece_id=ANL43CYDWEHDCQGXMYDNC5IOMA67TUFI7NEJWD2BQPKBBFM4UZRA, seg_pos={0 0}, upload=476901]: Uploading piece... Dec 28 07:18:10.512304017 (543h49m10.383387062s) [noise=true , address=145.79.227.36:4518 , piece_id=72XJZ23ZUVNLNCG6ZYJ6HAAFMCQ6L4OGLJ432X3XCXA22U3Y35MA, seg_pos={0 0}, upload=476901]: Uploading piece... Dec 28 07:18:10.512908097 (543h49m10.383991142s) [noise=true , address=145.79.227.31:4797 , piece_id=EPAV3WQS3ABWN4WLNHNKHS7LHKUNRABJA5QCVPDQU7L3TSG74ELQ, seg_pos={0 0}, upload=476901]: Uploading piece... Dec 28 07:18:10.513607109 (543h49m10.384690154s) [noise=true , address=145.79.227.15:4873 , piece_id=6XRRTEFTT7PMJNM3B6H7XJ5Z47ADKSXM57AO33NEBWP3NER4GNPA, seg_pos={0 0}, upload=476901]: Uploading piece... Dec 28 07:18:10.514257687 (543h49m10.385340732s) [noise=true , address=45.91.116.14:4191 , piece_id=YJXAKGOAMMFPCT6TVTD7Z3QQ2Q4JLLEYU5WZI2CT5AP2XLJ2UV6Q, seg_pos={0 0}, upload=476901]: Uploading piece... Dec 28 07:18:10.516353427 (543h49m10.387436472s) [noise=true , address=145.79.227.6:4258 , piece_id=G5U7M3MRUYFPLD2AQCLKDRRKV7GDA4PBTO6H7M52LX5S6I3NNEAQ, seg_pos={0 0}, upload=476901]: Uploading piece... Dec 28 07:18:10.518588684 (543h49m10.389671749s) [noise=true , address=145.79.227.40:4051 , piece_id=32IWXZ67FVYCEJAXYOI5IM6R6KDNHZVWW6VWV46AVCYO6Y3QD36A, seg_pos={0 0}, upload=476901]: Uploading piece... Dec 28 07:18:10.520573925 (543h49m10.39165698s) [noise=true , address=45.91.116.10:4082 , piece_id=DLFRP5UK5FOTW2IMP5PG5Q6J3SQ46BBUEDKJOOSVLOG3TWGUZXQA, seg_pos={0 0}, upload=476901]: Uploading piece... Dec 28 07:18:10.520789575 (543h49m10.39187262s) [noise=true , address=145.79.227.46:4602 , piece_id=CVTWR2UI3HZDUIESIWZPPQWUYRBQZ7Y2BPV6XMI62EUWAB2E4AJA, seg_pos={0 0}, upload=476901]: Uploading piece... Dec 28 07:18:10.522245217 (543h49m10.393328262s) [noise=true , address=45.91.116.13:4156 , piece_id=WAB3YW7QS2OMM62WN7BDWNYNAWQAUG35PJ3A7BFMWPZIACB7XKYQ, seg_pos={0 0}, upload=476901]: Uploading piece... Dec 28 07:18:10.523986352 (543h49m10.395069387s) [noise=true , address=145.79.227.48:4303 , piece_id=DDWVOO7AYTQAJ7PG5PFJP7HYWD2BK5LM5R6HV7CYDCVTUUQMPOKQ, seg_pos={0 0}, upload=476901]: Uploading piece... Dec 28 07:18:10.524484260 (543h49m10.395567345s) [noise=true , address=145.79.227.41:4086 , piece_id=TJWPLXY72BGZRWR5RVYVTJVFMDDS6MGTEWCBWEESBJCBUEQJJ65A, seg_pos={0 0}, upload=476901]: Uploading piece... Dec 28 07:18:10.526174088 (543h49m10.397257133s) [noise=true , address=145.79.227.11:4731 , piece_id=3UIKIKSF6NLG2QZSYXCMID2NLTUIEMNT7CCLA3H3BAIEM53XXZVA, seg_pos={0 0}, upload=476901]: Uploading piece... Dec 28 07:18:10.527475967 (543h49m10.398559022s) [noise=true , address=145.79.227.45:4230 , piece_id=C4EHV5SVCJPHL7X2ZZ62MRXHIQYU7NSQF5MAHKT5F3WGCLQUIJJQ, seg_pos={0 0}, upload=476901]: Uploading piece... Dec 28 07:18:10.527943477 (543h49m10.399026512s) [noise=true , address=45.91.116.8:4047 , piece_id=XQFXT7QOWSK256QVXGG6WHXVOXBLTVCKYQXDRGMVOZGQMSU3U5KA, seg_pos={0 0}, upload=476901]: Uploading piece... Dec 28 07:18:12.240222444 (543h49m12.111305489s) [upload=476901]: Next returned nil segment Dec 28 07:18:12.240239546 (543h49m12.111322591s) [upload=476901]: Waiting for error group managing segments... Dec 28 07:18:12.527012666 (543h49m12.398095701s) [noise=true , address=145.79.227.48:4303 , piece_id=DDWVOO7AYTQAJ7PG5PFJP7HYWD2BK5LM5R6HV7CYDCVTUUQMPOKQ, seg_pos={0 0}, upload=476901]: Done uploading piece. err: <nil> Dec 28 07:18:12.531569833 (543h49m12.402652869s) [noise=true , address=145.79.227.42:4121 , piece_id=J3FKKSK7ILIFCR5E7OX3OZ7M5VUKESDFU7K2SOWAN6IWEFZYHHHQ, seg_pos={0 0}, upload=476901]: Done uploading piece. err: <nil> Dec 28 07:18:12.544112948 (543h49m12.415195983s) [noise=true , address=145.79.227.41:4086 , piece_id=TJWPLXY72BGZRWR5RVYVTJVFMDDS6MGTEWCBWEESBJCBUEQJJ65A, seg_pos={0 0}, upload=476901]: Done uploading piece. err: <nil> Dec 28 07:18:12.544526917 (543h49m12.415609962s) [noise=true , address=145.79.227.36:4518 , piece_id=72XJZ23ZUVNLNCG6ZYJ6HAAFMCQ6L4OGLJ432X3XCXA22U3Y35MA, seg_pos={0 0}, upload=476901]: Done uploading piece. err: <nil> Dec 28 07:18:12.548652943 (543h49m12.419735988s) [noise=true , address=145.79.227.15:4873 , piece_id=6XRRTEFTT7PMJNM3B6H7XJ5Z47ADKSXM57AO33NEBWP3NER4GNPA, seg_pos={0 0}, upload=476901]: Done uploading piece. err: <nil> Dec 28 07:18:12.556841916 (543h49m12.427924961s) [noise=true , address=145.79.227.6:4258 , piece_id=G5U7M3MRUYFPLD2AQCLKDRRKV7GDA4PBTO6H7M52LX5S6I3NNEAQ, seg_pos={0 0}, upload=476901]: Done uploading piece. err: <nil> Dec 28 07:18:12.562320609 (543h49m12.433403654s) [noise=true , address=145.79.227.45:4230 , piece_id=C4EHV5SVCJPHL7X2ZZ62MRXHIQYU7NSQF5MAHKT5F3WGCLQUIJJQ, seg_pos={0 0}, upload=476901]: Done uploading piece. err: <nil> Dec 28 07:18:12.565597108 (543h49m12.436680143s) [noise=true , address=145.79.227.46:4602 , piece_id=CVTWR2UI3HZDUIESIWZPPQWUYRBQZ7Y2BPV6XMI62EUWAB2E4AJA, seg_pos={0 0}, upload=476901]: Done uploading piece. err: <nil> Dec 28 07:18:12.567182076 (543h49m12.438265111s) [noise=true , address=145.79.227.31:4797 , piece_id=EPAV3WQS3ABWN4WLNHNKHS7LHKUNRABJA5QCVPDQU7L3TSG74ELQ, seg_pos={0 0}, upload=476901]: Done uploading piece. err: <nil> Dec 28 07:18:12.573387182 (543h49m12.444470227s) [noise=true , address=145.79.227.11:4731 , piece_id=3UIKIKSF6NLG2QZSYXCMID2NLTUIEMNT7CCLA3H3BAIEM53XXZVA, seg_pos={0 0}, upload=476901]: Done uploading piece. err: <nil> Dec 28 07:18:12.774457327 (543h49m12.645540372s) [noise=true , address=45.91.116.14:4191 , piece_id=YJXAKGOAMMFPCT6TVTD7Z3QQ2Q4JLLEYU5WZI2CT5AP2XLJ2UV6Q, seg_pos={0 0}, upload=476901]: Done uploading piece. err: <nil> Dec 28 07:18:12.819072252 (543h49m12.690155297s) [noise=true , address=45.91.116.7:4011 , piece_id=ANL43CYDWEHDCQGXMYDNC5IOMA67TUFI7NEJWD2BQPKBBFM4UZRA, seg_pos={0 0}, upload=476901]: Done uploading piece. err: <nil> Dec 28 07:18:12.860750024 (543h49m12.731833069s) [noise=true , address=45.91.116.10:4082 , piece_id=DLFRP5UK5FOTW2IMP5PG5Q6J3SQ46BBUEDKJOOSVLOG3TWGUZXQA, seg_pos={0 0}, upload=476901]: Done uploading piece. err: <nil> Dec 28 07:18:12.860773398 (543h49m12.731856433s) [seg_pos={0 0}, upload=476901]: Segment reached optimal threshold of 13 pieces. Dec 28 07:18:12.860962669 (543h49m12.732045714s) [noise=true , address=145.79.227.40:4051 , piece_id=32IWXZ67FVYCEJAXYOI5IM6R6KDNHZVWW6VWV46AVCYO6Y3QD36A, seg_pos={0 0}, upload=476901]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n): protocol: expected piece hash; context canceled; EOF Dec 28 07:18:12.861004048 (543h49m12.732087083s) [noise=true , address=145.79.227.40:4051 , piece_id=32IWXZ67FVYCEJAXYOI5IM6R6KDNHZVWW6VWV46AVCYO6Y3QD36A, seg_pos={0 0}, upload=476901]: Long tail cancellation due to optimal threshold Dec 28 07:18:12.861020769 (543h49m12.732103804s) [noise=true , address=45.91.116.8:4047 , piece_id=XQFXT7QOWSK256QVXGG6WHXVOXBLTVCKYQXDRGMVOZGQMSU3U5KA, seg_pos={0 0}, upload=476901]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: write tcp 188.166.225.170:55528->45.91.116.8:4047: use of closed network connection; write tcp 188.166.225.170:55528->45.91.116.8:4047: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55528->45.91.116.8:4047: use of closed network connection Dec 28 07:18:12.861029116 (543h49m12.732112151s) [noise=true , address=45.91.116.8:4047 , piece_id=XQFXT7QOWSK256QVXGG6WHXVOXBLTVCKYQXDRGMVOZGQMSU3U5KA, seg_pos={0 0}, upload=476901]: Long tail cancellation due to optimal threshold Dec 28 07:18:12.861140307 (543h49m12.732223352s) [noise=true , address=45.91.116.13:4156 , piece_id=WAB3YW7QS2OMM62WN7BDWNYNAWQAUG35PJ3A7BFMWPZIACB7XKYQ, seg_pos={0 0}, upload=476901]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: expected piece hash; context canceled; EOF Dec 28 07:18:12.861148122 (543h49m12.732231167s) [noise=true , address=45.91.116.13:4156 , piece_id=WAB3YW7QS2OMM62WN7BDWNYNAWQAUG35PJ3A7BFMWPZIACB7XKYQ, seg_pos={0 0}, upload=476901]: Long tail cancellation due to optimal threshold Dec 28 07:18:12.861170194 (543h49m12.732253239s) [seg_pos={0 0}, upload=476901]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 07:18:12.861265284 (543h49m12.73234833s) [upload=476901]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 07:18:12.861271457 (543h49m12.732354492s) [upload=476901]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 07:18:13.163324262 (543h49m13.034407297s) [upload=476901]: Done uploading segments... Dec 28 07:18:13.163344520 (543h49m13.034427565s) [upload=476901]: Upload finished. err: <nil> Duration: 5.739210428s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 23 out of 60 is failed (threshold=6) [..X.X.........X.X...XX......X.X..X..XX.XXXXXXX.X...X..X.XX..] duration: 5.000825968s trace: https://jaeger.dev.storj.io/trace/584f899091a23e15 FAIL (2025-12-28 07:19:51 UTC) orbiter warning: context deadline exceeded Duration: 5.000851457s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 23 out of 60 is failed (threshold=6) [...X.X.........X.X...XX......X.X..X..XX.XXXXXXX.X...X..X.XX.] duration: 5.000983266s trace: https://jaeger.dev.storj.io/trace/584f899091a23e15 FAIL (2025-12-28 07:19:33 UTC) orbiter warning: context deadline exceeded Duration: 5.001009498s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [.......X.XXX.XX.....X...XXX.XX.............XX.X...X...X.XXX.] duration: 5.911633644s trace: https://jaeger.dev.storj.io/trace/2fd38217318d19b1 FAIL (2025-12-28 07:19:15 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, Piece ID: DLFRP5UK5FOTW2IMP5PG5Q6J3SQ46BBUEDKJOOSVLOG3TWGUZXQA) context deadline exceeded; piecestore: (Node ID: 12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, Piece ID: ANL43CYDWEHDCQGXMYDNC5IOMA67TUFI7NEJWD2BQPKBBFM4UZRA) context deadline exceeded ======LOGS====== 8 07:18:10.252919992 (543h49m10.124003037s) [upload=476901]: Starting upload Dec 28 07:18:10.253074667 (543h49m10.124157702s) [upload=476901]: Done starting upload Dec 28 07:18:10.253165671 (543h49m10.124248726s) [upload=476901]: Uploading segments... Dec 28 07:18:10.253425986 (543h49m10.124509041s) [upload=476901]: Got next segment. Inline: false Dec 28 07:18:10.253469889 (543h49m10.124552924s) [upload=476901]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 07:18:10.253477404 (543h49m10.124560449s) [upload=476901]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 07:18:10.511603543 (543h49m10.382686598s) [seg_pos={0 0}, upload=476901]: Begin upload segment... Dec 28 07:18:10.511660832 (543h49m10.382743877s) [seg_pos={0 0}, upload=476901]: Done begin upload segment. Dec 28 07:18:10.511676382 (543h49m10.382759417s) [noise=true , address=145.79.227.42:4121 , piece_id=J3FKKSK7ILIFCR5E7OX3OZ7M5VUKESDFU7K2SOWAN6IWEFZYHHHQ, seg_pos={0 0}, upload=476901]: Uploading piece... Dec 28 07:18:10.511707311 (543h49m10.382790416s) [noise=true , address=45.91.116.7:4011 , piece_id=ANL43CYDWEHDCQGXMYDNC5IOMA67TUFI7NEJWD2BQPKBBFM4UZRA, seg_pos={0 0}, upload=476901]: Uploading piece... Dec 28 07:18:10.512304017 (543h49m10.383387062s) [noise=true , address=145.79.227.36:4518 , piece_id=72XJZ23ZUVNLNCG6ZYJ6HAAFMCQ6L4OGLJ432X3XCXA22U3Y35MA, seg_pos={0 0}, upload=476901]: Uploading piece... Dec 28 07:18:10.512908097 (543h49m10.383991142s) [noise=true , address=145.79.227.31:4797 , piece_id=EPAV3WQS3ABWN4WLNHNKHS7LHKUNRABJA5QCVPDQU7L3TSG74ELQ, seg_pos={0 0}, upload=476901]: Uploading piece... Dec 28 07:18:10.513607109 (543h49m10.384690154s) [noise=true , address=145.79.227.15:4873 , piece_id=6XRRTEFTT7PMJNM3B6H7XJ5Z47ADKSXM57AO33NEBWP3NER4GNPA, seg_pos={0 0}, upload=476901]: Uploading piece... Dec 28 07:18:10.514257687 (543h49m10.385340732s) [noise=true , address=45.91.116.14:4191 , piece_id=YJXAKGOAMMFPCT6TVTD7Z3QQ2Q4JLLEYU5WZI2CT5AP2XLJ2UV6Q, seg_pos={0 0}, upload=476901]: Uploading piece... Dec 28 07:18:10.516353427 (543h49m10.387436472s) [noise=true , address=145.79.227.6:4258 , piece_id=G5U7M3MRUYFPLD2AQCLKDRRKV7GDA4PBTO6H7M52LX5S6I3NNEAQ, seg_pos={0 0}, upload=476901]: Uploading piece... Dec 28 07:18:10.518588684 (543h49m10.389671749s) [noise=true , address=145.79.227.40:4051 , piece_id=32IWXZ67FVYCEJAXYOI5IM6R6KDNHZVWW6VWV46AVCYO6Y3QD36A, seg_pos={0 0}, upload=476901]: Uploading piece... Dec 28 07:18:10.520573925 (543h49m10.39165698s) [noise=true , address=45.91.116.10:4082 , piece_id=DLFRP5UK5FOTW2IMP5PG5Q6J3SQ46BBUEDKJOOSVLOG3TWGUZXQA, seg_pos={0 0}, upload=476901]: Uploading piece... Dec 28 07:18:10.520789575 (543h49m10.39187262s) [noise=true , address=145.79.227.46:4602 , piece_id=CVTWR2UI3HZDUIESIWZPPQWUYRBQZ7Y2BPV6XMI62EUWAB2E4AJA, seg_pos={0 0}, upload=476901]: Uploading piece... Dec 28 07:18:10.522245217 (543h49m10.393328262s) [noise=true , address=45.91.116.13:4156 , piece_id=WAB3YW7QS2OMM62WN7BDWNYNAWQAUG35PJ3A7BFMWPZIACB7XKYQ, seg_pos={0 0}, upload=476901]: Uploading piece... Dec 28 07:18:10.523986352 (543h49m10.395069387s) [noise=true , address=145.79.227.48:4303 , piece_id=DDWVOO7AYTQAJ7PG5PFJP7HYWD2BK5LM5R6HV7CYDCVTUUQMPOKQ, seg_pos={0 0}, upload=476901]: Uploading piece... Dec 28 07:18:10.524484260 (543h49m10.395567345s) [noise=true , address=145.79.227.41:4086 , piece_id=TJWPLXY72BGZRWR5RVYVTJVFMDDS6MGTEWCBWEESBJCBUEQJJ65A, seg_pos={0 0}, upload=476901]: Uploading piece... Dec 28 07:18:10.526174088 (543h49m10.397257133s) [noise=true , address=145.79.227.11:4731 , piece_id=3UIKIKSF6NLG2QZSYXCMID2NLTUIEMNT7CCLA3H3BAIEM53XXZVA, seg_pos={0 0}, upload=476901]: Uploading piece... Dec 28 07:18:10.527475967 (543h49m10.398559022s) [noise=true , address=145.79.227.45:4230 , piece_id=C4EHV5SVCJPHL7X2ZZ62MRXHIQYU7NSQF5MAHKT5F3WGCLQUIJJQ, seg_pos={0 0}, upload=476901]: Uploading piece... Dec 28 07:18:10.527943477 (543h49m10.399026512s) [noise=true , address=45.91.116.8:4047 , piece_id=XQFXT7QOWSK256QVXGG6WHXVOXBLTVCKYQXDRGMVOZGQMSU3U5KA, seg_pos={0 0}, upload=476901]: Uploading piece... Dec 28 07:18:12.240222444 (543h49m12.111305489s) [upload=476901]: Next returned nil segment Dec 28 07:18:12.240239546 (543h49m12.111322591s) [upload=476901]: Waiting for error group managing segments... Dec 28 07:18:12.527012666 (543h49m12.398095701s) [noise=true , address=145.79.227.48:4303 , piece_id=DDWVOO7AYTQAJ7PG5PFJP7HYWD2BK5LM5R6HV7CYDCVTUUQMPOKQ, seg_pos={0 0}, upload=476901]: Done uploading piece. err: <nil> Dec 28 07:18:12.531569833 (543h49m12.402652869s) [noise=true , address=145.79.227.42:4121 , piece_id=J3FKKSK7ILIFCR5E7OX3OZ7M5VUKESDFU7K2SOWAN6IWEFZYHHHQ, seg_pos={0 0}, upload=476901]: Done uploading piece. err: <nil> Dec 28 07:18:12.544112948 (543h49m12.415195983s) [noise=true , address=145.79.227.41:4086 , piece_id=TJWPLXY72BGZRWR5RVYVTJVFMDDS6MGTEWCBWEESBJCBUEQJJ65A, seg_pos={0 0}, upload=476901]: Done uploading piece. err: <nil> Dec 28 07:18:12.544526917 (543h49m12.415609962s) [noise=true , address=145.79.227.36:4518 , piece_id=72XJZ23ZUVNLNCG6ZYJ6HAAFMCQ6L4OGLJ432X3XCXA22U3Y35MA, seg_pos={0 0}, upload=476901]: Done uploading piece. err: <nil> Dec 28 07:18:12.548652943 (543h49m12.419735988s) [noise=true , address=145.79.227.15:4873 , piece_id=6XRRTEFTT7PMJNM3B6H7XJ5Z47ADKSXM57AO33NEBWP3NER4GNPA, seg_pos={0 0}, upload=476901]: Done uploading piece. err: <nil> Dec 28 07:18:12.556841916 (543h49m12.427924961s) [noise=true , address=145.79.227.6:4258 , piece_id=G5U7M3MRUYFPLD2AQCLKDRRKV7GDA4PBTO6H7M52LX5S6I3NNEAQ, seg_pos={0 0}, upload=476901]: Done uploading piece. err: <nil> Dec 28 07:18:12.562320609 (543h49m12.433403654s) [noise=true , address=145.79.227.45:4230 , piece_id=C4EHV5SVCJPHL7X2ZZ62MRXHIQYU7NSQF5MAHKT5F3WGCLQUIJJQ, seg_pos={0 0}, upload=476901]: Done uploading piece. err: <nil> Dec 28 07:18:12.565597108 (543h49m12.436680143s) [noise=true , address=145.79.227.46:4602 , piece_id=CVTWR2UI3HZDUIESIWZPPQWUYRBQZ7Y2BPV6XMI62EUWAB2E4AJA, seg_pos={0 0}, upload=476901]: Done uploading piece. err: <nil> Dec 28 07:18:12.567182076 (543h49m12.438265111s) [noise=true , address=145.79.227.31:4797 , piece_id=EPAV3WQS3ABWN4WLNHNKHS7LHKUNRABJA5QCVPDQU7L3TSG74ELQ, seg_pos={0 0}, upload=476901]: Done uploading piece. err: <nil> Dec 28 07:18:12.573387182 (543h49m12.444470227s) [noise=true , address=145.79.227.11:4731 , piece_id=3UIKIKSF6NLG2QZSYXCMID2NLTUIEMNT7CCLA3H3BAIEM53XXZVA, seg_pos={0 0}, upload=476901]: Done uploading piece. err: <nil> Dec 28 07:18:12.774457327 (543h49m12.645540372s) [noise=true , address=45.91.116.14:4191 , piece_id=YJXAKGOAMMFPCT6TVTD7Z3QQ2Q4JLLEYU5WZI2CT5AP2XLJ2UV6Q, seg_pos={0 0}, upload=476901]: Done uploading piece. err: <nil> Dec 28 07:18:12.819072252 (543h49m12.690155297s) [noise=true , address=45.91.116.7:4011 , piece_id=ANL43CYDWEHDCQGXMYDNC5IOMA67TUFI7NEJWD2BQPKBBFM4UZRA, seg_pos={0 0}, upload=476901]: Done uploading piece. err: <nil> Dec 28 07:18:12.860750024 (543h49m12.731833069s) [noise=true , address=45.91.116.10:4082 , piece_id=DLFRP5UK5FOTW2IMP5PG5Q6J3SQ46BBUEDKJOOSVLOG3TWGUZXQA, seg_pos={0 0}, upload=476901]: Done uploading piece. err: <nil> Dec 28 07:18:12.860773398 (543h49m12.731856433s) [seg_pos={0 0}, upload=476901]: Segment reached optimal threshold of 13 pieces. Dec 28 07:18:12.860962669 (543h49m12.732045714s) [noise=true , address=145.79.227.40:4051 , piece_id=32IWXZ67FVYCEJAXYOI5IM6R6KDNHZVWW6VWV46AVCYO6Y3QD36A, seg_pos={0 0}, upload=476901]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n): protocol: expected piece hash; context canceled; EOF Dec 28 07:18:12.861004048 (543h49m12.732087083s) [noise=true , address=145.79.227.40:4051 , piece_id=32IWXZ67FVYCEJAXYOI5IM6R6KDNHZVWW6VWV46AVCYO6Y3QD36A, seg_pos={0 0}, upload=476901]: Long tail cancellation due to optimal threshold Dec 28 07:18:12.861020769 (543h49m12.732103804s) [noise=true , address=45.91.116.8:4047 , piece_id=XQFXT7QOWSK256QVXGG6WHXVOXBLTVCKYQXDRGMVOZGQMSU3U5KA, seg_pos={0 0}, upload=476901]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: write tcp 188.166.225.170:55528->45.91.116.8:4047: use of closed network connection; write tcp 188.166.225.170:55528->45.91.116.8:4047: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55528->45.91.116.8:4047: use of closed network connection Dec 28 07:18:12.861029116 (543h49m12.732112151s) [noise=true , address=45.91.116.8:4047 , piece_id=XQFXT7QOWSK256QVXGG6WHXVOXBLTVCKYQXDRGMVOZGQMSU3U5KA, seg_pos={0 0}, upload=476901]: Long tail cancellation due to optimal threshold Dec 28 07:18:12.861140307 (543h49m12.732223352s) [noise=true , address=45.91.116.13:4156 , piece_id=WAB3YW7QS2OMM62WN7BDWNYNAWQAUG35PJ3A7BFMWPZIACB7XKYQ, seg_pos={0 0}, upload=476901]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: expected piece hash; context canceled; EOF Dec 28 07:18:12.861148122 (543h49m12.732231167s) [noise=true , address=45.91.116.13:4156 , piece_id=WAB3YW7QS2OMM62WN7BDWNYNAWQAUG35PJ3A7BFMWPZIACB7XKYQ, seg_pos={0 0}, upload=476901]: Long tail cancellation due to optimal threshold Dec 28 07:18:12.861170194 (543h49m12.732253239s) [seg_pos={0 0}, upload=476901]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 07:18:12.861265284 (543h49m12.73234833s) [upload=476901]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 07:18:12.861271457 (543h49m12.732354492s) [upload=476901]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 07:18:13.163324262 (543h49m13.034407297s) [upload=476901]: Done uploading segments... Dec 28 07:18:13.163344520 (543h49m13.034427565s) [upload=476901]: Upload finished. err: <nil> Duration: 5.911666376s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 23 out of 60 is failed (threshold=6) [....X.X.........X.X...XX......X.X..X..XX.XXXXXXX.X...X..X.XX] duration: 5.001722728s trace: https://jaeger.dev.storj.io/trace/584f899091a23e15 FAIL (2025-12-28 07:19:07 UTC) orbiter warning: context deadline exceeded Duration: 5.001764046s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)