Orbiter History

Orbiter: EU/US/AP | Code | Config | Playbook | Edge Deployment | Escalation Procedure

Orbiter Check History

Showing 50 of 355426 reports (Page 216 of 7109)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [....X.XX..X..X.XX...............X..X.X..X....X........XX....] duration: 5.656319486s trace: https://jaeger.dev.storj.io/trace/4a02b772b0018a96 FAIL (2025-12-30 13:07:54 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, Piece ID: OVD3QYJFN6RPTEPXFSZWSXF3J2YXCXZTUEWWYJJMYF4OZ4BRCZDQ) context deadline exceeded; piecestore: (Node ID: 12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, Piece ID: KBZYZPT6HFCWY36ZQK37N6XQNZVFNOMLAGRAPJAA76MYQ2KBSX3A) context deadline exceeded ======LOGS====== Dec 30 13:05:17.988231417 (597h36m17.859314452s) [upload=523833]: Starting upload Dec 30 13:05:17.988290189 (597h36m17.859373224s) [upload=523833]: Done starting upload Dec 30 13:05:17.988476343 (597h36m17.859559378s) [upload=523833]: Uploading segments... Dec 30 13:05:17.988522001 (597h36m17.859605036s) [upload=523833]: Got next segment. Inline: false Dec 30 13:05:17.988548350 (597h36m17.859631385s) [upload=523833]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 13:05:17.988552870 (597h36m17.859635925s) [upload=523833]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 13:05:18.235197488 (597h36m18.106280523s) [seg_pos={0 0}, upload=523833]: Begin upload segment... Dec 30 13:05:18.235244077 (597h36m18.106327112s) [seg_pos={0 0}, upload=523833]: Done begin upload segment. Dec 30 13:05:18.235285987 (597h36m18.106369022s) [noise=true , address=145.79.227.9:4366 , piece_id=3HO7WFYJ5V7QMA4ULDRZ2QUDFC3X3ARDR2JLL4SBEH3R3442LEEQ, seg_pos={0 0}, upload=523833]: Uploading piece... Dec 30 13:05:18.235320202 (597h36m18.106403247s) [noise=true , address=45.91.116.11:4117 , piece_id=CDYF2GSH5JXBJZ6U4MLTRZPKJV7JMZDOC7UXWRYLWVDLSYKRJK5Q, seg_pos={0 0}, upload=523833]: Uploading piece... Dec 30 13:05:18.235847166 (597h36m18.106930201s) [noise=true , address=145.79.227.3:4151 , piece_id=4SPC56GTTPAUOEQS6WIYKT4XS5I7TH4NJTNL2SH44CUB4AGC7S5Q, seg_pos={0 0}, upload=523833]: Uploading piece... Dec 30 13:05:18.235886951 (597h36m18.106969996s) [noise=true , address=145.79.227.2:4115 , piece_id=Y3ZWB3BFPATJ3NUY7OUVPUO3KJ7QSURWXP6EGJY4D6CRLIYJSZ3Q, seg_pos={0 0}, upload=523833]: Uploading piece... Dec 30 13:05:18.236164309 (597h36m18.107247344s) [noise=true , address=77.90.31.13:4626 , piece_id=V27YSE6V3NJYGPHWOXJSJUNXNWXFPUX3TEOK4B7VXSFIA26Y5CPQ, seg_pos={0 0}, upload=523833]: Uploading piece... Dec 30 13:05:18.236275291 (597h36m18.107358336s) [noise=true , address=145.79.227.25:4617 , piece_id=3IPLSEQ4QUYBTD3J5RGBRVBB7MJ2BPJZ2SJWPLX2GULITVPHEZVQ, seg_pos={0 0}, upload=523833]: Uploading piece... Dec 30 13:05:18.236525807 (597h36m18.107608842s) [noise=true , address=145.79.227.32:4834 , piece_id=6CVFHJDNVJXYCFQSKYJ4JGE45AFLLUZJBKG53GF7TGPOPUBG2YLQ, seg_pos={0 0}, upload=523833]: Uploading piece... Dec 30 13:05:18.236699468 (597h36m18.107782503s) [noise=true , address=145.79.227.50:4375 , piece_id=BYGKORNIBEJSJYBLZTWF2JHW4DC65I4O5WQG2AUU6AZBU2NVTEVA, seg_pos={0 0}, upload=523833]: Uploading piece... Dec 30 13:05:18.236797896 (597h36m18.107880931s) [noise=true , address=145.79.227.52:4447 , piece_id=4UTPOVLC4FIAGMAGULQDRCKKS4FDSADZ36A62VHQ7Y5YJKWKSNZQ, seg_pos={0 0}, upload=523833]: Uploading piece... Dec 30 13:05:18.237062679 (597h36m18.108145714s) [noise=true , address=145.79.227.34:4905 , piece_id=J32TZMXC5FM3FPJM674QCXWHRLN2KQT2F6KYNCM3KA5GO4UY4CHA, seg_pos={0 0}, upload=523833]: Uploading piece... Dec 30 13:05:18.237424669 (597h36m18.108507704s) [noise=true , address=77.90.31.21:4877 , piece_id=OVD3QYJFN6RPTEPXFSZWSXF3J2YXCXZTUEWWYJJMYF4OZ4BRCZDQ, seg_pos={0 0}, upload=523833]: Uploading piece... Dec 30 13:05:18.237720462 (597h36m18.108803497s) [noise=true , address=77.90.31.15:4697 , piece_id=KBZYZPT6HFCWY36ZQK37N6XQNZVFNOMLAGRAPJAA76MYQ2KBSX3A, seg_pos={0 0}, upload=523833]: Uploading piece... Dec 30 13:05:18.238346965 (597h36m18.10943s) [noise=true , address=145.79.227.46:4602 , piece_id=YG7Y3JZELTLE2NM2LLDLRXB3MU2TQKXZGPTKWM5VNRJIZHHRO27Q, seg_pos={0 0}, upload=523833]: Uploading piece... Dec 30 13:05:18.238818193 (597h36m18.109901228s) [noise=true , address=145.79.227.31:4799 , piece_id=X6UQZM7YVLMCR5X35EGQ7N6RBGYDYRJMI2YH7DV3CAORBDUMFHAQ, seg_pos={0 0}, upload=523833]: Uploading piece... Dec 30 13:05:18.239195892 (597h36m18.110278927s) [noise=true , address=145.79.227.12:4767 , piece_id=I2U5XKOLG2WSPVXF7MGXXSGJQIFFSJ46LZLNLATI7EMGYWTACI6Q, seg_pos={0 0}, upload=523833]: Uploading piece... Dec 30 13:05:18.239419397 (597h36m18.110502442s) [noise=true , address=145.79.227.20:4438 , piece_id=G2LUF234AWZ5GIL6DTUIWVBGTPR6B6RZDVGKBSRULCNZJOXDVN5Q, seg_pos={0 0}, upload=523833]: Uploading piece... Dec 30 13:05:19.938524132 (597h36m19.809607167s) [upload=523833]: Next returned nil segment Dec 30 13:05:19.938538439 (597h36m19.809621474s) [upload=523833]: Waiting for error group managing segments... Dec 30 13:05:20.225416789 (597h36m20.096499834s) [noise=true , address=145.79.227.46:4602 , piece_id=YG7Y3JZELTLE2NM2LLDLRXB3MU2TQKXZGPTKWM5VNRJIZHHRO27Q, seg_pos={0 0}, upload=523833]: Done uploading piece. err: <nil> Dec 30 13:05:20.230259981 (597h36m20.101343026s) [noise=true , address=145.79.227.3:4151 , piece_id=4SPC56GTTPAUOEQS6WIYKT4XS5I7TH4NJTNL2SH44CUB4AGC7S5Q, seg_pos={0 0}, upload=523833]: Done uploading piece. err: <nil> Dec 30 13:05:20.231961621 (597h36m20.103044666s) [noise=true , address=145.79.227.20:4438 , piece_id=G2LUF234AWZ5GIL6DTUIWVBGTPR6B6RZDVGKBSRULCNZJOXDVN5Q, seg_pos={0 0}, upload=523833]: Done uploading piece. err: <nil> Dec 30 13:05:20.236052922 (597h36m20.107135967s) [noise=true , address=145.79.227.52:4447 , piece_id=4UTPOVLC4FIAGMAGULQDRCKKS4FDSADZ36A62VHQ7Y5YJKWKSNZQ, seg_pos={0 0}, upload=523833]: Done uploading piece. err: <nil> Dec 30 13:05:20.245330678 (597h36m20.116413723s) [noise=true , address=145.79.227.9:4366 , piece_id=3HO7WFYJ5V7QMA4ULDRZ2QUDFC3X3ARDR2JLL4SBEH3R3442LEEQ, seg_pos={0 0}, upload=523833]: Done uploading piece. err: <nil> Dec 30 13:05:20.246275477 (597h36m20.117358512s) [noise=true , address=145.79.227.25:4617 , piece_id=3IPLSEQ4QUYBTD3J5RGBRVBB7MJ2BPJZ2SJWPLX2GULITVPHEZVQ, seg_pos={0 0}, upload=523833]: Done uploading piece. err: <nil> Dec 30 13:05:20.246765681 (597h36m20.117848726s) [noise=true , address=145.79.227.2:4115 , piece_id=Y3ZWB3BFPATJ3NUY7OUVPUO3KJ7QSURWXP6EGJY4D6CRLIYJSZ3Q, seg_pos={0 0}, upload=523833]: Done uploading piece. err: <nil> Dec 30 13:05:20.247080049 (597h36m20.118163084s) [noise=true , address=145.79.227.32:4834 , piece_id=6CVFHJDNVJXYCFQSKYJ4JGE45AFLLUZJBKG53GF7TGPOPUBG2YLQ, seg_pos={0 0}, upload=523833]: Done uploading piece. err: <nil> Dec 30 13:05:20.271725659 (597h36m20.142808704s) [noise=true , address=145.79.227.31:4799 , piece_id=X6UQZM7YVLMCR5X35EGQ7N6RBGYDYRJMI2YH7DV3CAORBDUMFHAQ, seg_pos={0 0}, upload=523833]: Done uploading piece. err: <nil> Dec 30 13:05:20.306214921 (597h36m20.177297956s) [noise=true , address=145.79.227.12:4767 , piece_id=I2U5XKOLG2WSPVXF7MGXXSGJQIFFSJ46LZLNLATI7EMGYWTACI6Q, seg_pos={0 0}, upload=523833]: Done uploading piece. err: <nil> Dec 30 13:05:20.311072280 (597h36m20.182155325s) [noise=true , address=145.79.227.50:4375 , piece_id=BYGKORNIBEJSJYBLZTWF2JHW4DC65I4O5WQG2AUU6AZBU2NVTEVA, seg_pos={0 0}, upload=523833]: Done uploading piece. err: <nil> Dec 30 13:05:20.394910160 (597h36m20.265993206s) [noise=true , address=77.90.31.15:4697 , piece_id=KBZYZPT6HFCWY36ZQK37N6XQNZVFNOMLAGRAPJAA76MYQ2KBSX3A, seg_pos={0 0}, upload=523833]: Done uploading piece. err: <nil> Dec 30 13:05:20.484099100 (597h36m20.355182136s) [noise=true , address=77.90.31.21:4877 , piece_id=OVD3QYJFN6RPTEPXFSZWSXF3J2YXCXZTUEWWYJJMYF4OZ4BRCZDQ, seg_pos={0 0}, upload=523833]: Done uploading piece. err: <nil> Dec 30 13:05:20.484177910 (597h36m20.355260965s) [seg_pos={0 0}, upload=523833]: Segment reached optimal threshold of 13 pieces. Dec 30 13:05:20.484451902 (597h36m20.355534947s) [noise=true , address=45.91.116.11:4117 , piece_id=CDYF2GSH5JXBJZ6U4MLTRZPKJV7JMZDOC7UXWRYLWVDLSYKRJK5Q, seg_pos={0 0}, upload=523833]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6): protocol: expected piece hash; context canceled; EOF Dec 30 13:05:20.484461801 (597h36m20.355544836s) [noise=true , address=45.91.116.11:4117 , piece_id=CDYF2GSH5JXBJZ6U4MLTRZPKJV7JMZDOC7UXWRYLWVDLSYKRJK5Q, seg_pos={0 0}, upload=523833]: Long tail cancellation due to optimal threshold Dec 30 13:05:20.484631704 (597h36m20.355714749s) [noise=true , address=145.79.227.34:4905 , piece_id=J32TZMXC5FM3FPJM674QCXWHRLN2KQT2F6KYNCM3KA5GO4UY4CHA, seg_pos={0 0}, upload=523833]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ffKESNSVpXQzbwJinP6vvgE6zoF3RFz2NRUuuJHvd7bvk1gc9): protocol: expected piece hash; context canceled; EOF Dec 30 13:05:20.484643737 (597h36m20.355726772s) [noise=true , address=145.79.227.34:4905 , piece_id=J32TZMXC5FM3FPJM674QCXWHRLN2KQT2F6KYNCM3KA5GO4UY4CHA, seg_pos={0 0}, upload=523833]: Long tail cancellation due to optimal threshold Dec 30 13:05:20.484847886 (597h36m20.355930931s) [noise=true , address=77.90.31.13:4626 , piece_id=V27YSE6V3NJYGPHWOXJSJUNXNWXFPUX3TEOK4B7VXSFIA26Y5CPQ, seg_pos={0 0}, upload=523833]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu): protocol: context canceled; context canceled Dec 30 13:05:20.484857505 (597h36m20.35594054s) [noise=true , address=77.90.31.13:4626 , piece_id=V27YSE6V3NJYGPHWOXJSJUNXNWXFPUX3TEOK4B7VXSFIA26Y5CPQ, seg_pos={0 0}, upload=523833]: Long tail cancellation due to optimal threshold Dec 30 13:05:20.484881309 (597h36m20.355964355s) [seg_pos={0 0}, upload=523833]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 13:05:20.484989906 (597h36m20.356072951s) [upload=523833]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 13:05:20.484997742 (597h36m20.356080777s) [upload=523833]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 13:05:20.779558830 (597h36m20.650641885s) [upload=523833]: Done uploading segments... Dec 30 13:05:20.779584489 (597h36m20.650667544s) [upload=523833]: Upload finished. err: <nil> Duration: 5.656340987s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [..........X.......................X....X....................] duration: 28.691797072s trace: https://jaeger.dev.storj.io/trace/7b47a373389cd7c3 FAIL (2025-12-30 13:07:52 UTC) orbiter warning: context deadline exceeded Duration: 28.691823723s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 8 out of 60 is failed (threshold=3) [....X.....X.X...X.........X....X........X..............X....] duration: 29.314596566s trace: https://jaeger.dev.storj.io/trace/59feccf4c7c3ba3a FAIL (2025-12-30 13:07:51 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: quiescence ======LOGS====== ec 30 13:04:06.775086282 (597h35m6.646169327s) [noise=true , address=145.79.227.40:4076 , piece_id=XGXIJP6AWHH322IXAUG3QEOO2AEPBEO6E52X6VGC226CAZUKQ7NA, seg_pos={0 0}, upload=523818]: Uploading piece... Dec 30 13:04:06.775516872 (597h35m6.646599907s) [noise=true , address=77.90.31.15:4708 , piece_id=K6SXMEM4SWGSWASJT46GOND2LF4REXOJ2RZJCPCGGQU7JJ2VGKDA, seg_pos={0 0}, upload=523818]: Uploading piece... Dec 30 13:04:06.775911253 (597h35m6.646994298s) [noise=true , address=64.49.13.27:4964 , piece_id=RFCAZJ226BWPK37KCUOIJ7UQ2UJLDQQ6YILNZO67FEONKRA5LB6A, seg_pos={0 0}, upload=523818]: Uploading piece... Dec 30 13:04:06.776366340 (597h35m6.647449395s) [noise=true , address=45.91.116.14:4222 , piece_id=4JWMPKJRZPU2FDGXCS5JW5CP6OL3T3XLNGEUTXEGSDBBKHMUFIOA, seg_pos={0 0}, upload=523818]: Uploading piece... Dec 30 13:04:06.777866967 (597h35m6.648950012s) [noise=true , address=145.79.227.34:4936 , piece_id=K3HBCI7M3WQIJ3GMWMQN33F3UQPWUMQW33NT5H52BNKDD7QNCQRQ, seg_pos={0 0}, upload=523818]: Uploading piece... Dec 30 13:04:06.778495334 (597h35m6.649578389s) [noise=true , address=145.79.227.5:4253 , piece_id=LOBQRRK5BLWDTAPUJB42YT3NC2G6M2J7EU3ALUMLTRD77NPYYFLQ, seg_pos={0 0}, upload=523818]: Uploading piece... Dec 30 13:04:06.779041454 (597h35m6.650124499s) [noise=true , address=77.90.31.18:4823 , piece_id=CZMTHJYNM5JF6VOFS7FTNABHNIWJTUTPHFRYYR63E2NNIKKCTWEQ, seg_pos={0 0}, upload=523818]: Uploading piece... Dec 30 13:04:06.779558509 (597h35m6.650641554s) [noise=true , address=45.91.116.8:4061 , piece_id=G5N5V7ISYQYRIPRSQI6YKO6HESVJOIF7N4GDYY7UJCPCUAZUER6A, seg_pos={0 0}, upload=523818]: Uploading piece... Dec 30 13:04:06.780053321 (597h35m6.651136366s) [noise=true , address=77.90.31.18:4836 , piece_id=NQ3OA633Y5TGCVGP5QCDHVG762ERUB5MB5GDLTCXIAX5S5XLJUQA, seg_pos={0 0}, upload=523818]: Uploading piece... Dec 30 13:04:06.780550918 (597h35m6.651633963s) [noise=true , address=77.90.31.12:4597 , piece_id=YMNNNTOZWOQ2VYT4VWVW7VHIUCQKZ6GMQMAYSUEQAEISXYO4SKRA, seg_pos={0 0}, upload=523818]: Uploading piece... Dec 30 13:04:06.781069085 (597h35m6.65215213s) [noise=true , address=145.79.227.51:4413 , piece_id=IGKBEDGRF5ZTI7UUPASHF5IFZWJSJZ2V6ATPHWTNW3SRMDSPRCIA, seg_pos={0 0}, upload=523818]: Uploading piece... Dec 30 13:04:06.781978246 (597h35m6.653061291s) [noise=true , address=145.79.227.49:4344 , piece_id=Z6LDJHQ6KLS3O6HDUIJI5EAGKH3Y7ZYIKFTXSGRVTTBOTVUCSRSQ, seg_pos={0 0}, upload=523818]: Uploading piece... Dec 30 13:04:06.782926150 (597h35m6.654009195s) [noise=true , address=145.79.227.44:4226 , piece_id=UPBKK6FMKBJTSNLDVHWP7BBVRMIHESW3ED7CQLUMEXRZGD7LYO5A, seg_pos={0 0}, upload=523818]: Uploading piece... Dec 30 13:04:06.783851333 (597h35m6.654934388s) [noise=true , address=45.91.116.9:4289 , piece_id=PFVRYB6YFVSYFD4ZSS7CDLORUXSG3VGG2SE4ZNE5YBFQTRL5MEJQ, seg_pos={0 0}, upload=523818]: Uploading piece... Dec 30 13:04:06.784805539 (597h35m6.655888584s) [noise=true , address=193.149.164.143:4703 , piece_id=WFBUJIEKQUXYFSSVB27M2YEEILDNN55E5KL23EUCUC2JDR3Z7XSQ, seg_pos={0 0}, upload=523818]: Uploading piece... Dec 30 13:04:06.786955854 (597h35m6.658038899s) [noise=true , address=145.79.227.30:4793 , piece_id=VWYFBUFM4NKIK7G7UXVUYAFZT3YYUW42W65PKWE4QKF7L2IO4YYA, seg_pos={0 0}, upload=523818]: Uploading piece... Dec 30 13:04:08.223121485 (597h35m8.09420453s) [upload=523818]: Next returned nil segment Dec 30 13:04:08.223139148 (597h35m8.094222193s) [upload=523818]: Waiting for error group managing segments... Dec 30 13:04:08.717719473 (597h35m8.588802518s) [noise=true , address=145.79.227.5:4253 , piece_id=LOBQRRK5BLWDTAPUJB42YT3NC2G6M2J7EU3ALUMLTRD77NPYYFLQ, seg_pos={0 0}, upload=523818]: Done uploading piece. err: <nil> Dec 30 13:04:08.717831707 (597h35m8.588914752s) [noise=true , address=145.79.227.51:4413 , piece_id=IGKBEDGRF5ZTI7UUPASHF5IFZWJSJZ2V6ATPHWTNW3SRMDSPRCIA, seg_pos={0 0}, upload=523818]: Done uploading piece. err: <nil> Dec 30 13:04:08.743536652 (597h35m8.614619687s) [noise=true , address=145.79.227.43:4187 , piece_id=NQCXR5HGRYEQ3WW27PQJLMUOEEUXZDGGQJQQO5NYGHPXPAQFWG7A, seg_pos={0 0}, upload=523818]: Done uploading piece. err: <nil> Dec 30 13:04:08.848591082 (597h35m8.719674127s) [noise=true , address=77.90.31.15:4708 , piece_id=K6SXMEM4SWGSWASJT46GOND2LF4REXOJ2RZJCPCGGQU7JJ2VGKDA, seg_pos={0 0}, upload=523818]: Done uploading piece. err: <nil> Dec 30 13:04:08.899414969 (597h35m8.770498014s) [noise=true , address=64.49.13.27:4964 , piece_id=RFCAZJ226BWPK37KCUOIJ7UQ2UJLDQQ6YILNZO67FEONKRA5LB6A, seg_pos={0 0}, upload=523818]: Done uploading piece. err: <nil> Dec 30 13:04:09.123129336 (597h35m8.994212371s) [noise=true , address=77.90.31.18:4836 , piece_id=NQ3OA633Y5TGCVGP5QCDHVG762ERUB5MB5GDLTCXIAX5S5XLJUQA, seg_pos={0 0}, upload=523818]: Done uploading piece. err: <nil> Dec 30 13:04:09.211969381 (597h35m9.083052416s) [noise=true , address=45.91.116.8:4061 , piece_id=G5N5V7ISYQYRIPRSQI6YKO6HESVJOIF7N4GDYY7UJCPCUAZUER6A, seg_pos={0 0}, upload=523818]: Done uploading piece. err: <nil> Dec 30 13:04:09.294556076 (597h35m9.165639142s) [noise=true , address=77.90.31.12:4597 , piece_id=YMNNNTOZWOQ2VYT4VWVW7VHIUCQKZ6GMQMAYSUEQAEISXYO4SKRA, seg_pos={0 0}, upload=523818]: Done uploading piece. err: <nil> Dec 30 13:04:09.479068837 (597h35m9.350151872s) [noise=true , address=193.149.164.143:4703 , piece_id=WFBUJIEKQUXYFSSVB27M2YEEILDNN55E5KL23EUCUC2JDR3Z7XSQ, seg_pos={0 0}, upload=523818]: Done uploading piece. err: <nil> Dec 30 13:04:09.510445983 (597h35m9.381529018s) [noise=true , address=145.79.227.44:4226 , piece_id=UPBKK6FMKBJTSNLDVHWP7BBVRMIHESW3ED7CQLUMEXRZGD7LYO5A, seg_pos={0 0}, upload=523818]: Done uploading piece. err: <nil> Dec 30 13:04:09.536218769 (597h35m9.407301814s) [noise=true , address=145.79.227.30:4793 , piece_id=VWYFBUFM4NKIK7G7UXVUYAFZT3YYUW42W65PKWE4QKF7L2IO4YYA, seg_pos={0 0}, upload=523818]: Done uploading piece. err: <nil> Dec 30 13:04:09.566859122 (597h35m9.437942167s) [noise=true , address=145.79.227.33:4877 , piece_id=BMMH4UJO7OQRDWK4QSSNCA3FVEUYXTX3TPWO62WMUG6DMQ5VTBEA, seg_pos={0 0}, upload=523818]: Done uploading piece. err: <nil> Dec 30 13:04:09.619658661 (597h35m9.490741696s) [noise=true , address=145.79.227.49:4344 , piece_id=Z6LDJHQ6KLS3O6HDUIJI5EAGKH3Y7ZYIKFTXSGRVTTBOTVUCSRSQ, seg_pos={0 0}, upload=523818]: Done uploading piece. err: <nil> Dec 30 13:04:09.621672355 (597h35m9.49275539s) [noise=true , address=45.91.116.14:4222 , piece_id=4JWMPKJRZPU2FDGXCS5JW5CP6OL3T3XLNGEUTXEGSDBBKHMUFIOA, seg_pos={0 0}, upload=523818]: Done uploading piece. err: <nil> Dec 30 13:04:09.621683898 (597h35m9.492766943s) [seg_pos={0 0}, upload=523818]: Segment reached optimal threshold of 14 pieces. Dec 30 13:04:09.621901051 (597h35m9.492984086s) [noise=true , address=77.90.31.18:4823 , piece_id=CZMTHJYNM5JF6VOFS7FTNABHNIWJTUTPHFRYYR63E2NNIKKCTWEQ, seg_pos={0 0}, upload=523818]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12uUCUDKZDwj25UcEKQ4WiTuQFMRbWw2oymGAdUdMX8wcUSd7jS): protocol: expected piece hash; context canceled; EOF Dec 30 13:04:09.621910259 (597h35m9.492993294s) [noise=true , address=77.90.31.18:4823 , piece_id=CZMTHJYNM5JF6VOFS7FTNABHNIWJTUTPHFRYYR63E2NNIKKCTWEQ, seg_pos={0 0}, upload=523818]: Long tail cancellation due to optimal threshold Dec 30 13:04:09.622063460 (597h35m9.493146505s) [noise=true , address=45.91.116.9:4289 , piece_id=PFVRYB6YFVSYFD4ZSS7CDLORUXSG3VGG2SE4ZNE5YBFQTRL5MEJQ, seg_pos={0 0}, upload=523818]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rfBAoVRdhYiRJFWqnamoRAFLxEUsk7tShJEG3jvhKDbcHwipD): protocol: write tcp 188.166.225.170:33424->45.91.116.9:4289: use of closed network connection; write tcp 188.166.225.170:33424->45.91.116.9:4289: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33424->45.91.116.9:4289: use of closed network connection Dec 30 13:04:09.622087205 (597h35m9.49317025s) [noise=true , address=45.91.116.9:4289 , piece_id=PFVRYB6YFVSYFD4ZSS7CDLORUXSG3VGG2SE4ZNE5YBFQTRL5MEJQ, seg_pos={0 0}, upload=523818]: Long tail cancellation due to optimal threshold Dec 30 13:04:09.622135457 (597h35m9.493218492s) [noise=true , address=145.79.227.40:4076 , piece_id=XGXIJP6AWHH322IXAUG3QEOO2AEPBEO6E52X6VGC226CAZUKQ7NA, seg_pos={0 0}, upload=523818]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19KofWeQV4jMQzmSCv3LUsUqAqGfAuDqyo9GGsqKGi7ctT3FA2): protocol: expected piece hash; context canceled; EOF Dec 30 13:04:09.622142100 (597h35m9.493225135s) [noise=true , address=145.79.227.40:4076 , piece_id=XGXIJP6AWHH322IXAUG3QEOO2AEPBEO6E52X6VGC226CAZUKQ7NA, seg_pos={0 0}, upload=523818]: Long tail cancellation due to optimal threshold Dec 30 13:04:09.622242572 (597h35m9.493325607s) [noise=true , address=145.79.227.34:4936 , piece_id=K3HBCI7M3WQIJ3GMWMQN33F3UQPWUMQW33NT5H52BNKDD7QNCQRQ, seg_pos={0 0}, upload=523818]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DicJh1hd5NUoj9uxoqzeaCmjGHwfk2VUaA9zQFmv72wBdBoW8): protocol: expected piece hash; context canceled; EOF Dec 30 13:04:09.622247882 (597h35m9.493330927s) [noise=true , address=145.79.227.34:4936 , piece_id=K3HBCI7M3WQIJ3GMWMQN33F3UQPWUMQW33NT5H52BNKDD7QNCQRQ, seg_pos={0 0}, upload=523818]: Long tail cancellation due to optimal threshold Dec 30 13:04:09.622337151 (597h35m9.493420186s) [seg_pos={0 0}, upload=523818]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 13:04:09.622456739 (597h35m9.493539795s) [upload=523818]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 13:04:09.622466388 (597h35m9.493549433s) [upload=523818]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 13:04:09.936998074 (597h35m9.808081119s) [upload=523818]: Done uploading segments... Dec 30 13:04:09.937018172 (597h35m9.808101217s) [upload=523818]: Upload finished. err: <nil> Duration: 29.314623078s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [.....X.XX..X..X.XX...............X..X.X..X....X........XX...] duration: 8.043661479s trace: https://jaeger.dev.storj.io/trace/4a02b772b0018a96 FAIL (2025-12-30 13:07:36 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, Piece ID: OVD3QYJFN6RPTEPXFSZWSXF3J2YXCXZTUEWWYJJMYF4OZ4BRCZDQ) context deadline exceeded; piecestore: (Node ID: 12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, Piece ID: KBZYZPT6HFCWY36ZQK37N6XQNZVFNOMLAGRAPJAA76MYQ2KBSX3A) context deadline exceeded ======LOGS====== Dec 30 13:05:17.988231417 (597h36m17.859314452s) [upload=523833]: Starting upload Dec 30 13:05:17.988290189 (597h36m17.859373224s) [upload=523833]: Done starting upload Dec 30 13:05:17.988476343 (597h36m17.859559378s) [upload=523833]: Uploading segments... Dec 30 13:05:17.988522001 (597h36m17.859605036s) [upload=523833]: Got next segment. Inline: false Dec 30 13:05:17.988548350 (597h36m17.859631385s) [upload=523833]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 13:05:17.988552870 (597h36m17.859635925s) [upload=523833]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 13:05:18.235197488 (597h36m18.106280523s) [seg_pos={0 0}, upload=523833]: Begin upload segment... Dec 30 13:05:18.235244077 (597h36m18.106327112s) [seg_pos={0 0}, upload=523833]: Done begin upload segment. Dec 30 13:05:18.235285987 (597h36m18.106369022s) [noise=true , address=145.79.227.9:4366 , piece_id=3HO7WFYJ5V7QMA4ULDRZ2QUDFC3X3ARDR2JLL4SBEH3R3442LEEQ, seg_pos={0 0}, upload=523833]: Uploading piece... Dec 30 13:05:18.235320202 (597h36m18.106403247s) [noise=true , address=45.91.116.11:4117 , piece_id=CDYF2GSH5JXBJZ6U4MLTRZPKJV7JMZDOC7UXWRYLWVDLSYKRJK5Q, seg_pos={0 0}, upload=523833]: Uploading piece... Dec 30 13:05:18.235847166 (597h36m18.106930201s) [noise=true , address=145.79.227.3:4151 , piece_id=4SPC56GTTPAUOEQS6WIYKT4XS5I7TH4NJTNL2SH44CUB4AGC7S5Q, seg_pos={0 0}, upload=523833]: Uploading piece... Dec 30 13:05:18.235886951 (597h36m18.106969996s) [noise=true , address=145.79.227.2:4115 , piece_id=Y3ZWB3BFPATJ3NUY7OUVPUO3KJ7QSURWXP6EGJY4D6CRLIYJSZ3Q, seg_pos={0 0}, upload=523833]: Uploading piece... Dec 30 13:05:18.236164309 (597h36m18.107247344s) [noise=true , address=77.90.31.13:4626 , piece_id=V27YSE6V3NJYGPHWOXJSJUNXNWXFPUX3TEOK4B7VXSFIA26Y5CPQ, seg_pos={0 0}, upload=523833]: Uploading piece... Dec 30 13:05:18.236275291 (597h36m18.107358336s) [noise=true , address=145.79.227.25:4617 , piece_id=3IPLSEQ4QUYBTD3J5RGBRVBB7MJ2BPJZ2SJWPLX2GULITVPHEZVQ, seg_pos={0 0}, upload=523833]: Uploading piece... Dec 30 13:05:18.236525807 (597h36m18.107608842s) [noise=true , address=145.79.227.32:4834 , piece_id=6CVFHJDNVJXYCFQSKYJ4JGE45AFLLUZJBKG53GF7TGPOPUBG2YLQ, seg_pos={0 0}, upload=523833]: Uploading piece... Dec 30 13:05:18.236699468 (597h36m18.107782503s) [noise=true , address=145.79.227.50:4375 , piece_id=BYGKORNIBEJSJYBLZTWF2JHW4DC65I4O5WQG2AUU6AZBU2NVTEVA, seg_pos={0 0}, upload=523833]: Uploading piece... Dec 30 13:05:18.236797896 (597h36m18.107880931s) [noise=true , address=145.79.227.52:4447 , piece_id=4UTPOVLC4FIAGMAGULQDRCKKS4FDSADZ36A62VHQ7Y5YJKWKSNZQ, seg_pos={0 0}, upload=523833]: Uploading piece... Dec 30 13:05:18.237062679 (597h36m18.108145714s) [noise=true , address=145.79.227.34:4905 , piece_id=J32TZMXC5FM3FPJM674QCXWHRLN2KQT2F6KYNCM3KA5GO4UY4CHA, seg_pos={0 0}, upload=523833]: Uploading piece... Dec 30 13:05:18.237424669 (597h36m18.108507704s) [noise=true , address=77.90.31.21:4877 , piece_id=OVD3QYJFN6RPTEPXFSZWSXF3J2YXCXZTUEWWYJJMYF4OZ4BRCZDQ, seg_pos={0 0}, upload=523833]: Uploading piece... Dec 30 13:05:18.237720462 (597h36m18.108803497s) [noise=true , address=77.90.31.15:4697 , piece_id=KBZYZPT6HFCWY36ZQK37N6XQNZVFNOMLAGRAPJAA76MYQ2KBSX3A, seg_pos={0 0}, upload=523833]: Uploading piece... Dec 30 13:05:18.238346965 (597h36m18.10943s) [noise=true , address=145.79.227.46:4602 , piece_id=YG7Y3JZELTLE2NM2LLDLRXB3MU2TQKXZGPTKWM5VNRJIZHHRO27Q, seg_pos={0 0}, upload=523833]: Uploading piece... Dec 30 13:05:18.238818193 (597h36m18.109901228s) [noise=true , address=145.79.227.31:4799 , piece_id=X6UQZM7YVLMCR5X35EGQ7N6RBGYDYRJMI2YH7DV3CAORBDUMFHAQ, seg_pos={0 0}, upload=523833]: Uploading piece... Dec 30 13:05:18.239195892 (597h36m18.110278927s) [noise=true , address=145.79.227.12:4767 , piece_id=I2U5XKOLG2WSPVXF7MGXXSGJQIFFSJ46LZLNLATI7EMGYWTACI6Q, seg_pos={0 0}, upload=523833]: Uploading piece... Dec 30 13:05:18.239419397 (597h36m18.110502442s) [noise=true , address=145.79.227.20:4438 , piece_id=G2LUF234AWZ5GIL6DTUIWVBGTPR6B6RZDVGKBSRULCNZJOXDVN5Q, seg_pos={0 0}, upload=523833]: Uploading piece... Dec 30 13:05:19.938524132 (597h36m19.809607167s) [upload=523833]: Next returned nil segment Dec 30 13:05:19.938538439 (597h36m19.809621474s) [upload=523833]: Waiting for error group managing segments... Dec 30 13:05:20.225416789 (597h36m20.096499834s) [noise=true , address=145.79.227.46:4602 , piece_id=YG7Y3JZELTLE2NM2LLDLRXB3MU2TQKXZGPTKWM5VNRJIZHHRO27Q, seg_pos={0 0}, upload=523833]: Done uploading piece. err: <nil> Dec 30 13:05:20.230259981 (597h36m20.101343026s) [noise=true , address=145.79.227.3:4151 , piece_id=4SPC56GTTPAUOEQS6WIYKT4XS5I7TH4NJTNL2SH44CUB4AGC7S5Q, seg_pos={0 0}, upload=523833]: Done uploading piece. err: <nil> Dec 30 13:05:20.231961621 (597h36m20.103044666s) [noise=true , address=145.79.227.20:4438 , piece_id=G2LUF234AWZ5GIL6DTUIWVBGTPR6B6RZDVGKBSRULCNZJOXDVN5Q, seg_pos={0 0}, upload=523833]: Done uploading piece. err: <nil> Dec 30 13:05:20.236052922 (597h36m20.107135967s) [noise=true , address=145.79.227.52:4447 , piece_id=4UTPOVLC4FIAGMAGULQDRCKKS4FDSADZ36A62VHQ7Y5YJKWKSNZQ, seg_pos={0 0}, upload=523833]: Done uploading piece. err: <nil> Dec 30 13:05:20.245330678 (597h36m20.116413723s) [noise=true , address=145.79.227.9:4366 , piece_id=3HO7WFYJ5V7QMA4ULDRZ2QUDFC3X3ARDR2JLL4SBEH3R3442LEEQ, seg_pos={0 0}, upload=523833]: Done uploading piece. err: <nil> Dec 30 13:05:20.246275477 (597h36m20.117358512s) [noise=true , address=145.79.227.25:4617 , piece_id=3IPLSEQ4QUYBTD3J5RGBRVBB7MJ2BPJZ2SJWPLX2GULITVPHEZVQ, seg_pos={0 0}, upload=523833]: Done uploading piece. err: <nil> Dec 30 13:05:20.246765681 (597h36m20.117848726s) [noise=true , address=145.79.227.2:4115 , piece_id=Y3ZWB3BFPATJ3NUY7OUVPUO3KJ7QSURWXP6EGJY4D6CRLIYJSZ3Q, seg_pos={0 0}, upload=523833]: Done uploading piece. err: <nil> Dec 30 13:05:20.247080049 (597h36m20.118163084s) [noise=true , address=145.79.227.32:4834 , piece_id=6CVFHJDNVJXYCFQSKYJ4JGE45AFLLUZJBKG53GF7TGPOPUBG2YLQ, seg_pos={0 0}, upload=523833]: Done uploading piece. err: <nil> Dec 30 13:05:20.271725659 (597h36m20.142808704s) [noise=true , address=145.79.227.31:4799 , piece_id=X6UQZM7YVLMCR5X35EGQ7N6RBGYDYRJMI2YH7DV3CAORBDUMFHAQ, seg_pos={0 0}, upload=523833]: Done uploading piece. err: <nil> Dec 30 13:05:20.306214921 (597h36m20.177297956s) [noise=true , address=145.79.227.12:4767 , piece_id=I2U5XKOLG2WSPVXF7MGXXSGJQIFFSJ46LZLNLATI7EMGYWTACI6Q, seg_pos={0 0}, upload=523833]: Done uploading piece. err: <nil> Dec 30 13:05:20.311072280 (597h36m20.182155325s) [noise=true , address=145.79.227.50:4375 , piece_id=BYGKORNIBEJSJYBLZTWF2JHW4DC65I4O5WQG2AUU6AZBU2NVTEVA, seg_pos={0 0}, upload=523833]: Done uploading piece. err: <nil> Dec 30 13:05:20.394910160 (597h36m20.265993206s) [noise=true , address=77.90.31.15:4697 , piece_id=KBZYZPT6HFCWY36ZQK37N6XQNZVFNOMLAGRAPJAA76MYQ2KBSX3A, seg_pos={0 0}, upload=523833]: Done uploading piece. err: <nil> Dec 30 13:05:20.484099100 (597h36m20.355182136s) [noise=true , address=77.90.31.21:4877 , piece_id=OVD3QYJFN6RPTEPXFSZWSXF3J2YXCXZTUEWWYJJMYF4OZ4BRCZDQ, seg_pos={0 0}, upload=523833]: Done uploading piece. err: <nil> Dec 30 13:05:20.484177910 (597h36m20.355260965s) [seg_pos={0 0}, upload=523833]: Segment reached optimal threshold of 13 pieces. Dec 30 13:05:20.484451902 (597h36m20.355534947s) [noise=true , address=45.91.116.11:4117 , piece_id=CDYF2GSH5JXBJZ6U4MLTRZPKJV7JMZDOC7UXWRYLWVDLSYKRJK5Q, seg_pos={0 0}, upload=523833]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6): protocol: expected piece hash; context canceled; EOF Dec 30 13:05:20.484461801 (597h36m20.355544836s) [noise=true , address=45.91.116.11:4117 , piece_id=CDYF2GSH5JXBJZ6U4MLTRZPKJV7JMZDOC7UXWRYLWVDLSYKRJK5Q, seg_pos={0 0}, upload=523833]: Long tail cancellation due to optimal threshold Dec 30 13:05:20.484631704 (597h36m20.355714749s) [noise=true , address=145.79.227.34:4905 , piece_id=J32TZMXC5FM3FPJM674QCXWHRLN2KQT2F6KYNCM3KA5GO4UY4CHA, seg_pos={0 0}, upload=523833]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ffKESNSVpXQzbwJinP6vvgE6zoF3RFz2NRUuuJHvd7bvk1gc9): protocol: expected piece hash; context canceled; EOF Dec 30 13:05:20.484643737 (597h36m20.355726772s) [noise=true , address=145.79.227.34:4905 , piece_id=J32TZMXC5FM3FPJM674QCXWHRLN2KQT2F6KYNCM3KA5GO4UY4CHA, seg_pos={0 0}, upload=523833]: Long tail cancellation due to optimal threshold Dec 30 13:05:20.484847886 (597h36m20.355930931s) [noise=true , address=77.90.31.13:4626 , piece_id=V27YSE6V3NJYGPHWOXJSJUNXNWXFPUX3TEOK4B7VXSFIA26Y5CPQ, seg_pos={0 0}, upload=523833]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu): protocol: context canceled; context canceled Dec 30 13:05:20.484857505 (597h36m20.35594054s) [noise=true , address=77.90.31.13:4626 , piece_id=V27YSE6V3NJYGPHWOXJSJUNXNWXFPUX3TEOK4B7VXSFIA26Y5CPQ, seg_pos={0 0}, upload=523833]: Long tail cancellation due to optimal threshold Dec 30 13:05:20.484881309 (597h36m20.355964355s) [seg_pos={0 0}, upload=523833]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 13:05:20.484989906 (597h36m20.356072951s) [upload=523833]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 13:05:20.484997742 (597h36m20.356080777s) [upload=523833]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 13:05:20.779558830 (597h36m20.650641885s) [upload=523833]: Done uploading segments... Dec 30 13:05:20.779584489 (597h36m20.650667544s) [upload=523833]: Upload finished. err: <nil> Duration: 8.043687247s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [.........................X........X...........X..X........X.] duration: 22.896147479s trace: https://jaeger.dev.storj.io/trace/4b0c02016560c2c0 FAIL (2025-12-30 13:07:36 UTC) orbiter warning: context deadline exceeded Duration: 22.896335056s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 47 out of 60 is failed (threshold=6) [XX...X.XXXXXXXXXXX.X.XXXXXXXXX.X.XXXXXX.X.XXX.XXXX.XXXXXXXX.] duration: 5.001604719s trace: https://jaeger.dev.storj.io/trace/1fa4e7429f6cc283 FAIL (2025-12-30 13:07:00 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001637401s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [......X.XX..X..X.XX...............X..X.X..X....X........XX..] duration: 22.93997306s trace: https://jaeger.dev.storj.io/trace/4a02b772b0018a96 FAIL (2025-12-30 13:06:54 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, Piece ID: OVD3QYJFN6RPTEPXFSZWSXF3J2YXCXZTUEWWYJJMYF4OZ4BRCZDQ) context deadline exceeded; piecestore: (Node ID: 12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, Piece ID: KBZYZPT6HFCWY36ZQK37N6XQNZVFNOMLAGRAPJAA76MYQ2KBSX3A) context deadline exceeded ======LOGS====== Dec 30 13:05:17.988231417 (597h36m17.859314452s) [upload=523833]: Starting upload Dec 30 13:05:17.988290189 (597h36m17.859373224s) [upload=523833]: Done starting upload Dec 30 13:05:17.988476343 (597h36m17.859559378s) [upload=523833]: Uploading segments... Dec 30 13:05:17.988522001 (597h36m17.859605036s) [upload=523833]: Got next segment. Inline: false Dec 30 13:05:17.988548350 (597h36m17.859631385s) [upload=523833]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 13:05:17.988552870 (597h36m17.859635925s) [upload=523833]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 13:05:18.235197488 (597h36m18.106280523s) [seg_pos={0 0}, upload=523833]: Begin upload segment... Dec 30 13:05:18.235244077 (597h36m18.106327112s) [seg_pos={0 0}, upload=523833]: Done begin upload segment. Dec 30 13:05:18.235285987 (597h36m18.106369022s) [noise=true , address=145.79.227.9:4366 , piece_id=3HO7WFYJ5V7QMA4ULDRZ2QUDFC3X3ARDR2JLL4SBEH3R3442LEEQ, seg_pos={0 0}, upload=523833]: Uploading piece... Dec 30 13:05:18.235320202 (597h36m18.106403247s) [noise=true , address=45.91.116.11:4117 , piece_id=CDYF2GSH5JXBJZ6U4MLTRZPKJV7JMZDOC7UXWRYLWVDLSYKRJK5Q, seg_pos={0 0}, upload=523833]: Uploading piece... Dec 30 13:05:18.235847166 (597h36m18.106930201s) [noise=true , address=145.79.227.3:4151 , piece_id=4SPC56GTTPAUOEQS6WIYKT4XS5I7TH4NJTNL2SH44CUB4AGC7S5Q, seg_pos={0 0}, upload=523833]: Uploading piece... Dec 30 13:05:18.235886951 (597h36m18.106969996s) [noise=true , address=145.79.227.2:4115 , piece_id=Y3ZWB3BFPATJ3NUY7OUVPUO3KJ7QSURWXP6EGJY4D6CRLIYJSZ3Q, seg_pos={0 0}, upload=523833]: Uploading piece... Dec 30 13:05:18.236164309 (597h36m18.107247344s) [noise=true , address=77.90.31.13:4626 , piece_id=V27YSE6V3NJYGPHWOXJSJUNXNWXFPUX3TEOK4B7VXSFIA26Y5CPQ, seg_pos={0 0}, upload=523833]: Uploading piece... Dec 30 13:05:18.236275291 (597h36m18.107358336s) [noise=true , address=145.79.227.25:4617 , piece_id=3IPLSEQ4QUYBTD3J5RGBRVBB7MJ2BPJZ2SJWPLX2GULITVPHEZVQ, seg_pos={0 0}, upload=523833]: Uploading piece... Dec 30 13:05:18.236525807 (597h36m18.107608842s) [noise=true , address=145.79.227.32:4834 , piece_id=6CVFHJDNVJXYCFQSKYJ4JGE45AFLLUZJBKG53GF7TGPOPUBG2YLQ, seg_pos={0 0}, upload=523833]: Uploading piece... Dec 30 13:05:18.236699468 (597h36m18.107782503s) [noise=true , address=145.79.227.50:4375 , piece_id=BYGKORNIBEJSJYBLZTWF2JHW4DC65I4O5WQG2AUU6AZBU2NVTEVA, seg_pos={0 0}, upload=523833]: Uploading piece... Dec 30 13:05:18.236797896 (597h36m18.107880931s) [noise=true , address=145.79.227.52:4447 , piece_id=4UTPOVLC4FIAGMAGULQDRCKKS4FDSADZ36A62VHQ7Y5YJKWKSNZQ, seg_pos={0 0}, upload=523833]: Uploading piece... Dec 30 13:05:18.237062679 (597h36m18.108145714s) [noise=true , address=145.79.227.34:4905 , piece_id=J32TZMXC5FM3FPJM674QCXWHRLN2KQT2F6KYNCM3KA5GO4UY4CHA, seg_pos={0 0}, upload=523833]: Uploading piece... Dec 30 13:05:18.237424669 (597h36m18.108507704s) [noise=true , address=77.90.31.21:4877 , piece_id=OVD3QYJFN6RPTEPXFSZWSXF3J2YXCXZTUEWWYJJMYF4OZ4BRCZDQ, seg_pos={0 0}, upload=523833]: Uploading piece... Dec 30 13:05:18.237720462 (597h36m18.108803497s) [noise=true , address=77.90.31.15:4697 , piece_id=KBZYZPT6HFCWY36ZQK37N6XQNZVFNOMLAGRAPJAA76MYQ2KBSX3A, seg_pos={0 0}, upload=523833]: Uploading piece... Dec 30 13:05:18.238346965 (597h36m18.10943s) [noise=true , address=145.79.227.46:4602 , piece_id=YG7Y3JZELTLE2NM2LLDLRXB3MU2TQKXZGPTKWM5VNRJIZHHRO27Q, seg_pos={0 0}, upload=523833]: Uploading piece... Dec 30 13:05:18.238818193 (597h36m18.109901228s) [noise=true , address=145.79.227.31:4799 , piece_id=X6UQZM7YVLMCR5X35EGQ7N6RBGYDYRJMI2YH7DV3CAORBDUMFHAQ, seg_pos={0 0}, upload=523833]: Uploading piece... Dec 30 13:05:18.239195892 (597h36m18.110278927s) [noise=true , address=145.79.227.12:4767 , piece_id=I2U5XKOLG2WSPVXF7MGXXSGJQIFFSJ46LZLNLATI7EMGYWTACI6Q, seg_pos={0 0}, upload=523833]: Uploading piece... Dec 30 13:05:18.239419397 (597h36m18.110502442s) [noise=true , address=145.79.227.20:4438 , piece_id=G2LUF234AWZ5GIL6DTUIWVBGTPR6B6RZDVGKBSRULCNZJOXDVN5Q, seg_pos={0 0}, upload=523833]: Uploading piece... Dec 30 13:05:19.938524132 (597h36m19.809607167s) [upload=523833]: Next returned nil segment Dec 30 13:05:19.938538439 (597h36m19.809621474s) [upload=523833]: Waiting for error group managing segments... Dec 30 13:05:20.225416789 (597h36m20.096499834s) [noise=true , address=145.79.227.46:4602 , piece_id=YG7Y3JZELTLE2NM2LLDLRXB3MU2TQKXZGPTKWM5VNRJIZHHRO27Q, seg_pos={0 0}, upload=523833]: Done uploading piece. err: <nil> Dec 30 13:05:20.230259981 (597h36m20.101343026s) [noise=true , address=145.79.227.3:4151 , piece_id=4SPC56GTTPAUOEQS6WIYKT4XS5I7TH4NJTNL2SH44CUB4AGC7S5Q, seg_pos={0 0}, upload=523833]: Done uploading piece. err: <nil> Dec 30 13:05:20.231961621 (597h36m20.103044666s) [noise=true , address=145.79.227.20:4438 , piece_id=G2LUF234AWZ5GIL6DTUIWVBGTPR6B6RZDVGKBSRULCNZJOXDVN5Q, seg_pos={0 0}, upload=523833]: Done uploading piece. err: <nil> Dec 30 13:05:20.236052922 (597h36m20.107135967s) [noise=true , address=145.79.227.52:4447 , piece_id=4UTPOVLC4FIAGMAGULQDRCKKS4FDSADZ36A62VHQ7Y5YJKWKSNZQ, seg_pos={0 0}, upload=523833]: Done uploading piece. err: <nil> Dec 30 13:05:20.245330678 (597h36m20.116413723s) [noise=true , address=145.79.227.9:4366 , piece_id=3HO7WFYJ5V7QMA4ULDRZ2QUDFC3X3ARDR2JLL4SBEH3R3442LEEQ, seg_pos={0 0}, upload=523833]: Done uploading piece. err: <nil> Dec 30 13:05:20.246275477 (597h36m20.117358512s) [noise=true , address=145.79.227.25:4617 , piece_id=3IPLSEQ4QUYBTD3J5RGBRVBB7MJ2BPJZ2SJWPLX2GULITVPHEZVQ, seg_pos={0 0}, upload=523833]: Done uploading piece. err: <nil> Dec 30 13:05:20.246765681 (597h36m20.117848726s) [noise=true , address=145.79.227.2:4115 , piece_id=Y3ZWB3BFPATJ3NUY7OUVPUO3KJ7QSURWXP6EGJY4D6CRLIYJSZ3Q, seg_pos={0 0}, upload=523833]: Done uploading piece. err: <nil> Dec 30 13:05:20.247080049 (597h36m20.118163084s) [noise=true , address=145.79.227.32:4834 , piece_id=6CVFHJDNVJXYCFQSKYJ4JGE45AFLLUZJBKG53GF7TGPOPUBG2YLQ, seg_pos={0 0}, upload=523833]: Done uploading piece. err: <nil> Dec 30 13:05:20.271725659 (597h36m20.142808704s) [noise=true , address=145.79.227.31:4799 , piece_id=X6UQZM7YVLMCR5X35EGQ7N6RBGYDYRJMI2YH7DV3CAORBDUMFHAQ, seg_pos={0 0}, upload=523833]: Done uploading piece. err: <nil> Dec 30 13:05:20.306214921 (597h36m20.177297956s) [noise=true , address=145.79.227.12:4767 , piece_id=I2U5XKOLG2WSPVXF7MGXXSGJQIFFSJ46LZLNLATI7EMGYWTACI6Q, seg_pos={0 0}, upload=523833]: Done uploading piece. err: <nil> Dec 30 13:05:20.311072280 (597h36m20.182155325s) [noise=true , address=145.79.227.50:4375 , piece_id=BYGKORNIBEJSJYBLZTWF2JHW4DC65I4O5WQG2AUU6AZBU2NVTEVA, seg_pos={0 0}, upload=523833]: Done uploading piece. err: <nil> Dec 30 13:05:20.394910160 (597h36m20.265993206s) [noise=true , address=77.90.31.15:4697 , piece_id=KBZYZPT6HFCWY36ZQK37N6XQNZVFNOMLAGRAPJAA76MYQ2KBSX3A, seg_pos={0 0}, upload=523833]: Done uploading piece. err: <nil> Dec 30 13:05:20.484099100 (597h36m20.355182136s) [noise=true , address=77.90.31.21:4877 , piece_id=OVD3QYJFN6RPTEPXFSZWSXF3J2YXCXZTUEWWYJJMYF4OZ4BRCZDQ, seg_pos={0 0}, upload=523833]: Done uploading piece. err: <nil> Dec 30 13:05:20.484177910 (597h36m20.355260965s) [seg_pos={0 0}, upload=523833]: Segment reached optimal threshold of 13 pieces. Dec 30 13:05:20.484451902 (597h36m20.355534947s) [noise=true , address=45.91.116.11:4117 , piece_id=CDYF2GSH5JXBJZ6U4MLTRZPKJV7JMZDOC7UXWRYLWVDLSYKRJK5Q, seg_pos={0 0}, upload=523833]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6): protocol: expected piece hash; context canceled; EOF Dec 30 13:05:20.484461801 (597h36m20.355544836s) [noise=true , address=45.91.116.11:4117 , piece_id=CDYF2GSH5JXBJZ6U4MLTRZPKJV7JMZDOC7UXWRYLWVDLSYKRJK5Q, seg_pos={0 0}, upload=523833]: Long tail cancellation due to optimal threshold Dec 30 13:05:20.484631704 (597h36m20.355714749s) [noise=true , address=145.79.227.34:4905 , piece_id=J32TZMXC5FM3FPJM674QCXWHRLN2KQT2F6KYNCM3KA5GO4UY4CHA, seg_pos={0 0}, upload=523833]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ffKESNSVpXQzbwJinP6vvgE6zoF3RFz2NRUuuJHvd7bvk1gc9): protocol: expected piece hash; context canceled; EOF Dec 30 13:05:20.484643737 (597h36m20.355726772s) [noise=true , address=145.79.227.34:4905 , piece_id=J32TZMXC5FM3FPJM674QCXWHRLN2KQT2F6KYNCM3KA5GO4UY4CHA, seg_pos={0 0}, upload=523833]: Long tail cancellation due to optimal threshold Dec 30 13:05:20.484847886 (597h36m20.355930931s) [noise=true , address=77.90.31.13:4626 , piece_id=V27YSE6V3NJYGPHWOXJSJUNXNWXFPUX3TEOK4B7VXSFIA26Y5CPQ, seg_pos={0 0}, upload=523833]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu): protocol: context canceled; context canceled Dec 30 13:05:20.484857505 (597h36m20.35594054s) [noise=true , address=77.90.31.13:4626 , piece_id=V27YSE6V3NJYGPHWOXJSJUNXNWXFPUX3TEOK4B7VXSFIA26Y5CPQ, seg_pos={0 0}, upload=523833]: Long tail cancellation due to optimal threshold Dec 30 13:05:20.484881309 (597h36m20.355964355s) [seg_pos={0 0}, upload=523833]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 13:05:20.484989906 (597h36m20.356072951s) [upload=523833]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 13:05:20.484997742 (597h36m20.356080777s) [upload=523833]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 13:05:20.779558830 (597h36m20.650641885s) [upload=523833]: Done uploading segments... Dec 30 13:05:20.779584489 (597h36m20.650667544s) [upload=523833]: Upload finished. err: <nil> Duration: 22.940001897s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [...........X.......................X....X...................] duration: 24.502528655s trace: https://jaeger.dev.storj.io/trace/7b47a373389cd7c3 FAIL (2025-12-30 13:06:49 UTC) orbiter warning: context deadline exceeded Duration: 24.502543955s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [..........................X........X...........X..X........X] duration: 30.0016215s trace: https://jaeger.dev.storj.io/trace/4b0c02016560c2c0 FAIL (2025-12-30 13:06:43 UTC) orbiter warning: context deadline exceeded Duration: 30.001645757s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 48 out of 60 is failed (threshold=6) [XXX...X.XXXXXXXXXXX.X.XXXXXXXXX.X.XXXXXX.X.XXX.XXXX.XXXXXXXX] duration: 5.001564486s trace: https://jaeger.dev.storj.io/trace/1fa4e7429f6cc283 FAIL (2025-12-30 13:06:42 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001598141s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 8 out of 60 is failed (threshold=3) [.....X.....X.X...X.........X....X........X..............X...] duration: 15.459212392s trace: https://jaeger.dev.storj.io/trace/59feccf4c7c3ba3a FAIL (2025-12-30 13:06:35 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: quiescence ======LOGS====== ec 30 13:04:06.775086282 (597h35m6.646169327s) [noise=true , address=145.79.227.40:4076 , piece_id=XGXIJP6AWHH322IXAUG3QEOO2AEPBEO6E52X6VGC226CAZUKQ7NA, seg_pos={0 0}, upload=523818]: Uploading piece... Dec 30 13:04:06.775516872 (597h35m6.646599907s) [noise=true , address=77.90.31.15:4708 , piece_id=K6SXMEM4SWGSWASJT46GOND2LF4REXOJ2RZJCPCGGQU7JJ2VGKDA, seg_pos={0 0}, upload=523818]: Uploading piece... Dec 30 13:04:06.775911253 (597h35m6.646994298s) [noise=true , address=64.49.13.27:4964 , piece_id=RFCAZJ226BWPK37KCUOIJ7UQ2UJLDQQ6YILNZO67FEONKRA5LB6A, seg_pos={0 0}, upload=523818]: Uploading piece... Dec 30 13:04:06.776366340 (597h35m6.647449395s) [noise=true , address=45.91.116.14:4222 , piece_id=4JWMPKJRZPU2FDGXCS5JW5CP6OL3T3XLNGEUTXEGSDBBKHMUFIOA, seg_pos={0 0}, upload=523818]: Uploading piece... Dec 30 13:04:06.777866967 (597h35m6.648950012s) [noise=true , address=145.79.227.34:4936 , piece_id=K3HBCI7M3WQIJ3GMWMQN33F3UQPWUMQW33NT5H52BNKDD7QNCQRQ, seg_pos={0 0}, upload=523818]: Uploading piece... Dec 30 13:04:06.778495334 (597h35m6.649578389s) [noise=true , address=145.79.227.5:4253 , piece_id=LOBQRRK5BLWDTAPUJB42YT3NC2G6M2J7EU3ALUMLTRD77NPYYFLQ, seg_pos={0 0}, upload=523818]: Uploading piece... Dec 30 13:04:06.779041454 (597h35m6.650124499s) [noise=true , address=77.90.31.18:4823 , piece_id=CZMTHJYNM5JF6VOFS7FTNABHNIWJTUTPHFRYYR63E2NNIKKCTWEQ, seg_pos={0 0}, upload=523818]: Uploading piece... Dec 30 13:04:06.779558509 (597h35m6.650641554s) [noise=true , address=45.91.116.8:4061 , piece_id=G5N5V7ISYQYRIPRSQI6YKO6HESVJOIF7N4GDYY7UJCPCUAZUER6A, seg_pos={0 0}, upload=523818]: Uploading piece... Dec 30 13:04:06.780053321 (597h35m6.651136366s) [noise=true , address=77.90.31.18:4836 , piece_id=NQ3OA633Y5TGCVGP5QCDHVG762ERUB5MB5GDLTCXIAX5S5XLJUQA, seg_pos={0 0}, upload=523818]: Uploading piece... Dec 30 13:04:06.780550918 (597h35m6.651633963s) [noise=true , address=77.90.31.12:4597 , piece_id=YMNNNTOZWOQ2VYT4VWVW7VHIUCQKZ6GMQMAYSUEQAEISXYO4SKRA, seg_pos={0 0}, upload=523818]: Uploading piece... Dec 30 13:04:06.781069085 (597h35m6.65215213s) [noise=true , address=145.79.227.51:4413 , piece_id=IGKBEDGRF5ZTI7UUPASHF5IFZWJSJZ2V6ATPHWTNW3SRMDSPRCIA, seg_pos={0 0}, upload=523818]: Uploading piece... Dec 30 13:04:06.781978246 (597h35m6.653061291s) [noise=true , address=145.79.227.49:4344 , piece_id=Z6LDJHQ6KLS3O6HDUIJI5EAGKH3Y7ZYIKFTXSGRVTTBOTVUCSRSQ, seg_pos={0 0}, upload=523818]: Uploading piece... Dec 30 13:04:06.782926150 (597h35m6.654009195s) [noise=true , address=145.79.227.44:4226 , piece_id=UPBKK6FMKBJTSNLDVHWP7BBVRMIHESW3ED7CQLUMEXRZGD7LYO5A, seg_pos={0 0}, upload=523818]: Uploading piece... Dec 30 13:04:06.783851333 (597h35m6.654934388s) [noise=true , address=45.91.116.9:4289 , piece_id=PFVRYB6YFVSYFD4ZSS7CDLORUXSG3VGG2SE4ZNE5YBFQTRL5MEJQ, seg_pos={0 0}, upload=523818]: Uploading piece... Dec 30 13:04:06.784805539 (597h35m6.655888584s) [noise=true , address=193.149.164.143:4703 , piece_id=WFBUJIEKQUXYFSSVB27M2YEEILDNN55E5KL23EUCUC2JDR3Z7XSQ, seg_pos={0 0}, upload=523818]: Uploading piece... Dec 30 13:04:06.786955854 (597h35m6.658038899s) [noise=true , address=145.79.227.30:4793 , piece_id=VWYFBUFM4NKIK7G7UXVUYAFZT3YYUW42W65PKWE4QKF7L2IO4YYA, seg_pos={0 0}, upload=523818]: Uploading piece... Dec 30 13:04:08.223121485 (597h35m8.09420453s) [upload=523818]: Next returned nil segment Dec 30 13:04:08.223139148 (597h35m8.094222193s) [upload=523818]: Waiting for error group managing segments... Dec 30 13:04:08.717719473 (597h35m8.588802518s) [noise=true , address=145.79.227.5:4253 , piece_id=LOBQRRK5BLWDTAPUJB42YT3NC2G6M2J7EU3ALUMLTRD77NPYYFLQ, seg_pos={0 0}, upload=523818]: Done uploading piece. err: <nil> Dec 30 13:04:08.717831707 (597h35m8.588914752s) [noise=true , address=145.79.227.51:4413 , piece_id=IGKBEDGRF5ZTI7UUPASHF5IFZWJSJZ2V6ATPHWTNW3SRMDSPRCIA, seg_pos={0 0}, upload=523818]: Done uploading piece. err: <nil> Dec 30 13:04:08.743536652 (597h35m8.614619687s) [noise=true , address=145.79.227.43:4187 , piece_id=NQCXR5HGRYEQ3WW27PQJLMUOEEUXZDGGQJQQO5NYGHPXPAQFWG7A, seg_pos={0 0}, upload=523818]: Done uploading piece. err: <nil> Dec 30 13:04:08.848591082 (597h35m8.719674127s) [noise=true , address=77.90.31.15:4708 , piece_id=K6SXMEM4SWGSWASJT46GOND2LF4REXOJ2RZJCPCGGQU7JJ2VGKDA, seg_pos={0 0}, upload=523818]: Done uploading piece. err: <nil> Dec 30 13:04:08.899414969 (597h35m8.770498014s) [noise=true , address=64.49.13.27:4964 , piece_id=RFCAZJ226BWPK37KCUOIJ7UQ2UJLDQQ6YILNZO67FEONKRA5LB6A, seg_pos={0 0}, upload=523818]: Done uploading piece. err: <nil> Dec 30 13:04:09.123129336 (597h35m8.994212371s) [noise=true , address=77.90.31.18:4836 , piece_id=NQ3OA633Y5TGCVGP5QCDHVG762ERUB5MB5GDLTCXIAX5S5XLJUQA, seg_pos={0 0}, upload=523818]: Done uploading piece. err: <nil> Dec 30 13:04:09.211969381 (597h35m9.083052416s) [noise=true , address=45.91.116.8:4061 , piece_id=G5N5V7ISYQYRIPRSQI6YKO6HESVJOIF7N4GDYY7UJCPCUAZUER6A, seg_pos={0 0}, upload=523818]: Done uploading piece. err: <nil> Dec 30 13:04:09.294556076 (597h35m9.165639142s) [noise=true , address=77.90.31.12:4597 , piece_id=YMNNNTOZWOQ2VYT4VWVW7VHIUCQKZ6GMQMAYSUEQAEISXYO4SKRA, seg_pos={0 0}, upload=523818]: Done uploading piece. err: <nil> Dec 30 13:04:09.479068837 (597h35m9.350151872s) [noise=true , address=193.149.164.143:4703 , piece_id=WFBUJIEKQUXYFSSVB27M2YEEILDNN55E5KL23EUCUC2JDR3Z7XSQ, seg_pos={0 0}, upload=523818]: Done uploading piece. err: <nil> Dec 30 13:04:09.510445983 (597h35m9.381529018s) [noise=true , address=145.79.227.44:4226 , piece_id=UPBKK6FMKBJTSNLDVHWP7BBVRMIHESW3ED7CQLUMEXRZGD7LYO5A, seg_pos={0 0}, upload=523818]: Done uploading piece. err: <nil> Dec 30 13:04:09.536218769 (597h35m9.407301814s) [noise=true , address=145.79.227.30:4793 , piece_id=VWYFBUFM4NKIK7G7UXVUYAFZT3YYUW42W65PKWE4QKF7L2IO4YYA, seg_pos={0 0}, upload=523818]: Done uploading piece. err: <nil> Dec 30 13:04:09.566859122 (597h35m9.437942167s) [noise=true , address=145.79.227.33:4877 , piece_id=BMMH4UJO7OQRDWK4QSSNCA3FVEUYXTX3TPWO62WMUG6DMQ5VTBEA, seg_pos={0 0}, upload=523818]: Done uploading piece. err: <nil> Dec 30 13:04:09.619658661 (597h35m9.490741696s) [noise=true , address=145.79.227.49:4344 , piece_id=Z6LDJHQ6KLS3O6HDUIJI5EAGKH3Y7ZYIKFTXSGRVTTBOTVUCSRSQ, seg_pos={0 0}, upload=523818]: Done uploading piece. err: <nil> Dec 30 13:04:09.621672355 (597h35m9.49275539s) [noise=true , address=45.91.116.14:4222 , piece_id=4JWMPKJRZPU2FDGXCS5JW5CP6OL3T3XLNGEUTXEGSDBBKHMUFIOA, seg_pos={0 0}, upload=523818]: Done uploading piece. err: <nil> Dec 30 13:04:09.621683898 (597h35m9.492766943s) [seg_pos={0 0}, upload=523818]: Segment reached optimal threshold of 14 pieces. Dec 30 13:04:09.621901051 (597h35m9.492984086s) [noise=true , address=77.90.31.18:4823 , piece_id=CZMTHJYNM5JF6VOFS7FTNABHNIWJTUTPHFRYYR63E2NNIKKCTWEQ, seg_pos={0 0}, upload=523818]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12uUCUDKZDwj25UcEKQ4WiTuQFMRbWw2oymGAdUdMX8wcUSd7jS): protocol: expected piece hash; context canceled; EOF Dec 30 13:04:09.621910259 (597h35m9.492993294s) [noise=true , address=77.90.31.18:4823 , piece_id=CZMTHJYNM5JF6VOFS7FTNABHNIWJTUTPHFRYYR63E2NNIKKCTWEQ, seg_pos={0 0}, upload=523818]: Long tail cancellation due to optimal threshold Dec 30 13:04:09.622063460 (597h35m9.493146505s) [noise=true , address=45.91.116.9:4289 , piece_id=PFVRYB6YFVSYFD4ZSS7CDLORUXSG3VGG2SE4ZNE5YBFQTRL5MEJQ, seg_pos={0 0}, upload=523818]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rfBAoVRdhYiRJFWqnamoRAFLxEUsk7tShJEG3jvhKDbcHwipD): protocol: write tcp 188.166.225.170:33424->45.91.116.9:4289: use of closed network connection; write tcp 188.166.225.170:33424->45.91.116.9:4289: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33424->45.91.116.9:4289: use of closed network connection Dec 30 13:04:09.622087205 (597h35m9.49317025s) [noise=true , address=45.91.116.9:4289 , piece_id=PFVRYB6YFVSYFD4ZSS7CDLORUXSG3VGG2SE4ZNE5YBFQTRL5MEJQ, seg_pos={0 0}, upload=523818]: Long tail cancellation due to optimal threshold Dec 30 13:04:09.622135457 (597h35m9.493218492s) [noise=true , address=145.79.227.40:4076 , piece_id=XGXIJP6AWHH322IXAUG3QEOO2AEPBEO6E52X6VGC226CAZUKQ7NA, seg_pos={0 0}, upload=523818]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19KofWeQV4jMQzmSCv3LUsUqAqGfAuDqyo9GGsqKGi7ctT3FA2): protocol: expected piece hash; context canceled; EOF Dec 30 13:04:09.622142100 (597h35m9.493225135s) [noise=true , address=145.79.227.40:4076 , piece_id=XGXIJP6AWHH322IXAUG3QEOO2AEPBEO6E52X6VGC226CAZUKQ7NA, seg_pos={0 0}, upload=523818]: Long tail cancellation due to optimal threshold Dec 30 13:04:09.622242572 (597h35m9.493325607s) [noise=true , address=145.79.227.34:4936 , piece_id=K3HBCI7M3WQIJ3GMWMQN33F3UQPWUMQW33NT5H52BNKDD7QNCQRQ, seg_pos={0 0}, upload=523818]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DicJh1hd5NUoj9uxoqzeaCmjGHwfk2VUaA9zQFmv72wBdBoW8): protocol: expected piece hash; context canceled; EOF Dec 30 13:04:09.622247882 (597h35m9.493330927s) [noise=true , address=145.79.227.34:4936 , piece_id=K3HBCI7M3WQIJ3GMWMQN33F3UQPWUMQW33NT5H52BNKDD7QNCQRQ, seg_pos={0 0}, upload=523818]: Long tail cancellation due to optimal threshold Dec 30 13:04:09.622337151 (597h35m9.493420186s) [seg_pos={0 0}, upload=523818]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 13:04:09.622456739 (597h35m9.493539795s) [upload=523818]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 13:04:09.622466388 (597h35m9.493549433s) [upload=523818]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 13:04:09.936998074 (597h35m9.808081119s) [upload=523818]: Done uploading segments... Dec 30 13:04:09.937018172 (597h35m9.808101217s) [upload=523818]: Upload finished. err: <nil> Duration: 15.459228592s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [............X.......................X....X..................] duration: 25.617283109s trace: https://jaeger.dev.storj.io/trace/7b47a373389cd7c3 FAIL (2025-12-30 13:06:24 UTC) orbiter warning: context deadline exceeded Duration: 25.617307547s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 8 out of 60 is failed (threshold=3) [......X.....X.X...X.........X....X........X..............X..] duration: 29.741956128s trace: https://jaeger.dev.storj.io/trace/59feccf4c7c3ba3a FAIL (2025-12-30 13:06:19 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: quiescence ======LOGS====== ec 30 13:04:06.775086282 (597h35m6.646169327s) [noise=true , address=145.79.227.40:4076 , piece_id=XGXIJP6AWHH322IXAUG3QEOO2AEPBEO6E52X6VGC226CAZUKQ7NA, seg_pos={0 0}, upload=523818]: Uploading piece... Dec 30 13:04:06.775516872 (597h35m6.646599907s) [noise=true , address=77.90.31.15:4708 , piece_id=K6SXMEM4SWGSWASJT46GOND2LF4REXOJ2RZJCPCGGQU7JJ2VGKDA, seg_pos={0 0}, upload=523818]: Uploading piece... Dec 30 13:04:06.775911253 (597h35m6.646994298s) [noise=true , address=64.49.13.27:4964 , piece_id=RFCAZJ226BWPK37KCUOIJ7UQ2UJLDQQ6YILNZO67FEONKRA5LB6A, seg_pos={0 0}, upload=523818]: Uploading piece... Dec 30 13:04:06.776366340 (597h35m6.647449395s) [noise=true , address=45.91.116.14:4222 , piece_id=4JWMPKJRZPU2FDGXCS5JW5CP6OL3T3XLNGEUTXEGSDBBKHMUFIOA, seg_pos={0 0}, upload=523818]: Uploading piece... Dec 30 13:04:06.777866967 (597h35m6.648950012s) [noise=true , address=145.79.227.34:4936 , piece_id=K3HBCI7M3WQIJ3GMWMQN33F3UQPWUMQW33NT5H52BNKDD7QNCQRQ, seg_pos={0 0}, upload=523818]: Uploading piece... Dec 30 13:04:06.778495334 (597h35m6.649578389s) [noise=true , address=145.79.227.5:4253 , piece_id=LOBQRRK5BLWDTAPUJB42YT3NC2G6M2J7EU3ALUMLTRD77NPYYFLQ, seg_pos={0 0}, upload=523818]: Uploading piece... Dec 30 13:04:06.779041454 (597h35m6.650124499s) [noise=true , address=77.90.31.18:4823 , piece_id=CZMTHJYNM5JF6VOFS7FTNABHNIWJTUTPHFRYYR63E2NNIKKCTWEQ, seg_pos={0 0}, upload=523818]: Uploading piece... Dec 30 13:04:06.779558509 (597h35m6.650641554s) [noise=true , address=45.91.116.8:4061 , piece_id=G5N5V7ISYQYRIPRSQI6YKO6HESVJOIF7N4GDYY7UJCPCUAZUER6A, seg_pos={0 0}, upload=523818]: Uploading piece... Dec 30 13:04:06.780053321 (597h35m6.651136366s) [noise=true , address=77.90.31.18:4836 , piece_id=NQ3OA633Y5TGCVGP5QCDHVG762ERUB5MB5GDLTCXIAX5S5XLJUQA, seg_pos={0 0}, upload=523818]: Uploading piece... Dec 30 13:04:06.780550918 (597h35m6.651633963s) [noise=true , address=77.90.31.12:4597 , piece_id=YMNNNTOZWOQ2VYT4VWVW7VHIUCQKZ6GMQMAYSUEQAEISXYO4SKRA, seg_pos={0 0}, upload=523818]: Uploading piece... Dec 30 13:04:06.781069085 (597h35m6.65215213s) [noise=true , address=145.79.227.51:4413 , piece_id=IGKBEDGRF5ZTI7UUPASHF5IFZWJSJZ2V6ATPHWTNW3SRMDSPRCIA, seg_pos={0 0}, upload=523818]: Uploading piece... Dec 30 13:04:06.781978246 (597h35m6.653061291s) [noise=true , address=145.79.227.49:4344 , piece_id=Z6LDJHQ6KLS3O6HDUIJI5EAGKH3Y7ZYIKFTXSGRVTTBOTVUCSRSQ, seg_pos={0 0}, upload=523818]: Uploading piece... Dec 30 13:04:06.782926150 (597h35m6.654009195s) [noise=true , address=145.79.227.44:4226 , piece_id=UPBKK6FMKBJTSNLDVHWP7BBVRMIHESW3ED7CQLUMEXRZGD7LYO5A, seg_pos={0 0}, upload=523818]: Uploading piece... Dec 30 13:04:06.783851333 (597h35m6.654934388s) [noise=true , address=45.91.116.9:4289 , piece_id=PFVRYB6YFVSYFD4ZSS7CDLORUXSG3VGG2SE4ZNE5YBFQTRL5MEJQ, seg_pos={0 0}, upload=523818]: Uploading piece... Dec 30 13:04:06.784805539 (597h35m6.655888584s) [noise=true , address=193.149.164.143:4703 , piece_id=WFBUJIEKQUXYFSSVB27M2YEEILDNN55E5KL23EUCUC2JDR3Z7XSQ, seg_pos={0 0}, upload=523818]: Uploading piece... Dec 30 13:04:06.786955854 (597h35m6.658038899s) [noise=true , address=145.79.227.30:4793 , piece_id=VWYFBUFM4NKIK7G7UXVUYAFZT3YYUW42W65PKWE4QKF7L2IO4YYA, seg_pos={0 0}, upload=523818]: Uploading piece... Dec 30 13:04:08.223121485 (597h35m8.09420453s) [upload=523818]: Next returned nil segment Dec 30 13:04:08.223139148 (597h35m8.094222193s) [upload=523818]: Waiting for error group managing segments... Dec 30 13:04:08.717719473 (597h35m8.588802518s) [noise=true , address=145.79.227.5:4253 , piece_id=LOBQRRK5BLWDTAPUJB42YT3NC2G6M2J7EU3ALUMLTRD77NPYYFLQ, seg_pos={0 0}, upload=523818]: Done uploading piece. err: <nil> Dec 30 13:04:08.717831707 (597h35m8.588914752s) [noise=true , address=145.79.227.51:4413 , piece_id=IGKBEDGRF5ZTI7UUPASHF5IFZWJSJZ2V6ATPHWTNW3SRMDSPRCIA, seg_pos={0 0}, upload=523818]: Done uploading piece. err: <nil> Dec 30 13:04:08.743536652 (597h35m8.614619687s) [noise=true , address=145.79.227.43:4187 , piece_id=NQCXR5HGRYEQ3WW27PQJLMUOEEUXZDGGQJQQO5NYGHPXPAQFWG7A, seg_pos={0 0}, upload=523818]: Done uploading piece. err: <nil> Dec 30 13:04:08.848591082 (597h35m8.719674127s) [noise=true , address=77.90.31.15:4708 , piece_id=K6SXMEM4SWGSWASJT46GOND2LF4REXOJ2RZJCPCGGQU7JJ2VGKDA, seg_pos={0 0}, upload=523818]: Done uploading piece. err: <nil> Dec 30 13:04:08.899414969 (597h35m8.770498014s) [noise=true , address=64.49.13.27:4964 , piece_id=RFCAZJ226BWPK37KCUOIJ7UQ2UJLDQQ6YILNZO67FEONKRA5LB6A, seg_pos={0 0}, upload=523818]: Done uploading piece. err: <nil> Dec 30 13:04:09.123129336 (597h35m8.994212371s) [noise=true , address=77.90.31.18:4836 , piece_id=NQ3OA633Y5TGCVGP5QCDHVG762ERUB5MB5GDLTCXIAX5S5XLJUQA, seg_pos={0 0}, upload=523818]: Done uploading piece. err: <nil> Dec 30 13:04:09.211969381 (597h35m9.083052416s) [noise=true , address=45.91.116.8:4061 , piece_id=G5N5V7ISYQYRIPRSQI6YKO6HESVJOIF7N4GDYY7UJCPCUAZUER6A, seg_pos={0 0}, upload=523818]: Done uploading piece. err: <nil> Dec 30 13:04:09.294556076 (597h35m9.165639142s) [noise=true , address=77.90.31.12:4597 , piece_id=YMNNNTOZWOQ2VYT4VWVW7VHIUCQKZ6GMQMAYSUEQAEISXYO4SKRA, seg_pos={0 0}, upload=523818]: Done uploading piece. err: <nil> Dec 30 13:04:09.479068837 (597h35m9.350151872s) [noise=true , address=193.149.164.143:4703 , piece_id=WFBUJIEKQUXYFSSVB27M2YEEILDNN55E5KL23EUCUC2JDR3Z7XSQ, seg_pos={0 0}, upload=523818]: Done uploading piece. err: <nil> Dec 30 13:04:09.510445983 (597h35m9.381529018s) [noise=true , address=145.79.227.44:4226 , piece_id=UPBKK6FMKBJTSNLDVHWP7BBVRMIHESW3ED7CQLUMEXRZGD7LYO5A, seg_pos={0 0}, upload=523818]: Done uploading piece. err: <nil> Dec 30 13:04:09.536218769 (597h35m9.407301814s) [noise=true , address=145.79.227.30:4793 , piece_id=VWYFBUFM4NKIK7G7UXVUYAFZT3YYUW42W65PKWE4QKF7L2IO4YYA, seg_pos={0 0}, upload=523818]: Done uploading piece. err: <nil> Dec 30 13:04:09.566859122 (597h35m9.437942167s) [noise=true , address=145.79.227.33:4877 , piece_id=BMMH4UJO7OQRDWK4QSSNCA3FVEUYXTX3TPWO62WMUG6DMQ5VTBEA, seg_pos={0 0}, upload=523818]: Done uploading piece. err: <nil> Dec 30 13:04:09.619658661 (597h35m9.490741696s) [noise=true , address=145.79.227.49:4344 , piece_id=Z6LDJHQ6KLS3O6HDUIJI5EAGKH3Y7ZYIKFTXSGRVTTBOTVUCSRSQ, seg_pos={0 0}, upload=523818]: Done uploading piece. err: <nil> Dec 30 13:04:09.621672355 (597h35m9.49275539s) [noise=true , address=45.91.116.14:4222 , piece_id=4JWMPKJRZPU2FDGXCS5JW5CP6OL3T3XLNGEUTXEGSDBBKHMUFIOA, seg_pos={0 0}, upload=523818]: Done uploading piece. err: <nil> Dec 30 13:04:09.621683898 (597h35m9.492766943s) [seg_pos={0 0}, upload=523818]: Segment reached optimal threshold of 14 pieces. Dec 30 13:04:09.621901051 (597h35m9.492984086s) [noise=true , address=77.90.31.18:4823 , piece_id=CZMTHJYNM5JF6VOFS7FTNABHNIWJTUTPHFRYYR63E2NNIKKCTWEQ, seg_pos={0 0}, upload=523818]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12uUCUDKZDwj25UcEKQ4WiTuQFMRbWw2oymGAdUdMX8wcUSd7jS): protocol: expected piece hash; context canceled; EOF Dec 30 13:04:09.621910259 (597h35m9.492993294s) [noise=true , address=77.90.31.18:4823 , piece_id=CZMTHJYNM5JF6VOFS7FTNABHNIWJTUTPHFRYYR63E2NNIKKCTWEQ, seg_pos={0 0}, upload=523818]: Long tail cancellation due to optimal threshold Dec 30 13:04:09.622063460 (597h35m9.493146505s) [noise=true , address=45.91.116.9:4289 , piece_id=PFVRYB6YFVSYFD4ZSS7CDLORUXSG3VGG2SE4ZNE5YBFQTRL5MEJQ, seg_pos={0 0}, upload=523818]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rfBAoVRdhYiRJFWqnamoRAFLxEUsk7tShJEG3jvhKDbcHwipD): protocol: write tcp 188.166.225.170:33424->45.91.116.9:4289: use of closed network connection; write tcp 188.166.225.170:33424->45.91.116.9:4289: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33424->45.91.116.9:4289: use of closed network connection Dec 30 13:04:09.622087205 (597h35m9.49317025s) [noise=true , address=45.91.116.9:4289 , piece_id=PFVRYB6YFVSYFD4ZSS7CDLORUXSG3VGG2SE4ZNE5YBFQTRL5MEJQ, seg_pos={0 0}, upload=523818]: Long tail cancellation due to optimal threshold Dec 30 13:04:09.622135457 (597h35m9.493218492s) [noise=true , address=145.79.227.40:4076 , piece_id=XGXIJP6AWHH322IXAUG3QEOO2AEPBEO6E52X6VGC226CAZUKQ7NA, seg_pos={0 0}, upload=523818]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19KofWeQV4jMQzmSCv3LUsUqAqGfAuDqyo9GGsqKGi7ctT3FA2): protocol: expected piece hash; context canceled; EOF Dec 30 13:04:09.622142100 (597h35m9.493225135s) [noise=true , address=145.79.227.40:4076 , piece_id=XGXIJP6AWHH322IXAUG3QEOO2AEPBEO6E52X6VGC226CAZUKQ7NA, seg_pos={0 0}, upload=523818]: Long tail cancellation due to optimal threshold Dec 30 13:04:09.622242572 (597h35m9.493325607s) [noise=true , address=145.79.227.34:4936 , piece_id=K3HBCI7M3WQIJ3GMWMQN33F3UQPWUMQW33NT5H52BNKDD7QNCQRQ, seg_pos={0 0}, upload=523818]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DicJh1hd5NUoj9uxoqzeaCmjGHwfk2VUaA9zQFmv72wBdBoW8): protocol: expected piece hash; context canceled; EOF Dec 30 13:04:09.622247882 (597h35m9.493330927s) [noise=true , address=145.79.227.34:4936 , piece_id=K3HBCI7M3WQIJ3GMWMQN33F3UQPWUMQW33NT5H52BNKDD7QNCQRQ, seg_pos={0 0}, upload=523818]: Long tail cancellation due to optimal threshold Dec 30 13:04:09.622337151 (597h35m9.493420186s) [seg_pos={0 0}, upload=523818]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 13:04:09.622456739 (597h35m9.493539795s) [upload=523818]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 13:04:09.622466388 (597h35m9.493549433s) [upload=523818]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 13:04:09.936998074 (597h35m9.808081119s) [upload=523818]: Done uploading segments... Dec 30 13:04:09.937018172 (597h35m9.808101217s) [upload=523818]: Upload finished. err: <nil> Duration: 29.741971848s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [.......X.XX..X..X.XX...............X..X.X..X....X........XX.] duration: 6.250116549s trace: https://jaeger.dev.storj.io/trace/4a02b772b0018a96 FAIL (2025-12-30 13:06:01 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, Piece ID: OVD3QYJFN6RPTEPXFSZWSXF3J2YXCXZTUEWWYJJMYF4OZ4BRCZDQ) context deadline exceeded; piecestore: (Node ID: 12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, Piece ID: KBZYZPT6HFCWY36ZQK37N6XQNZVFNOMLAGRAPJAA76MYQ2KBSX3A) context deadline exceeded ======LOGS====== Dec 30 13:05:17.988231417 (597h36m17.859314452s) [upload=523833]: Starting upload Dec 30 13:05:17.988290189 (597h36m17.859373224s) [upload=523833]: Done starting upload Dec 30 13:05:17.988476343 (597h36m17.859559378s) [upload=523833]: Uploading segments... Dec 30 13:05:17.988522001 (597h36m17.859605036s) [upload=523833]: Got next segment. Inline: false Dec 30 13:05:17.988548350 (597h36m17.859631385s) [upload=523833]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 13:05:17.988552870 (597h36m17.859635925s) [upload=523833]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 13:05:18.235197488 (597h36m18.106280523s) [seg_pos={0 0}, upload=523833]: Begin upload segment... Dec 30 13:05:18.235244077 (597h36m18.106327112s) [seg_pos={0 0}, upload=523833]: Done begin upload segment. Dec 30 13:05:18.235285987 (597h36m18.106369022s) [noise=true , address=145.79.227.9:4366 , piece_id=3HO7WFYJ5V7QMA4ULDRZ2QUDFC3X3ARDR2JLL4SBEH3R3442LEEQ, seg_pos={0 0}, upload=523833]: Uploading piece... Dec 30 13:05:18.235320202 (597h36m18.106403247s) [noise=true , address=45.91.116.11:4117 , piece_id=CDYF2GSH5JXBJZ6U4MLTRZPKJV7JMZDOC7UXWRYLWVDLSYKRJK5Q, seg_pos={0 0}, upload=523833]: Uploading piece... Dec 30 13:05:18.235847166 (597h36m18.106930201s) [noise=true , address=145.79.227.3:4151 , piece_id=4SPC56GTTPAUOEQS6WIYKT4XS5I7TH4NJTNL2SH44CUB4AGC7S5Q, seg_pos={0 0}, upload=523833]: Uploading piece... Dec 30 13:05:18.235886951 (597h36m18.106969996s) [noise=true , address=145.79.227.2:4115 , piece_id=Y3ZWB3BFPATJ3NUY7OUVPUO3KJ7QSURWXP6EGJY4D6CRLIYJSZ3Q, seg_pos={0 0}, upload=523833]: Uploading piece... Dec 30 13:05:18.236164309 (597h36m18.107247344s) [noise=true , address=77.90.31.13:4626 , piece_id=V27YSE6V3NJYGPHWOXJSJUNXNWXFPUX3TEOK4B7VXSFIA26Y5CPQ, seg_pos={0 0}, upload=523833]: Uploading piece... Dec 30 13:05:18.236275291 (597h36m18.107358336s) [noise=true , address=145.79.227.25:4617 , piece_id=3IPLSEQ4QUYBTD3J5RGBRVBB7MJ2BPJZ2SJWPLX2GULITVPHEZVQ, seg_pos={0 0}, upload=523833]: Uploading piece... Dec 30 13:05:18.236525807 (597h36m18.107608842s) [noise=true , address=145.79.227.32:4834 , piece_id=6CVFHJDNVJXYCFQSKYJ4JGE45AFLLUZJBKG53GF7TGPOPUBG2YLQ, seg_pos={0 0}, upload=523833]: Uploading piece... Dec 30 13:05:18.236699468 (597h36m18.107782503s) [noise=true , address=145.79.227.50:4375 , piece_id=BYGKORNIBEJSJYBLZTWF2JHW4DC65I4O5WQG2AUU6AZBU2NVTEVA, seg_pos={0 0}, upload=523833]: Uploading piece... Dec 30 13:05:18.236797896 (597h36m18.107880931s) [noise=true , address=145.79.227.52:4447 , piece_id=4UTPOVLC4FIAGMAGULQDRCKKS4FDSADZ36A62VHQ7Y5YJKWKSNZQ, seg_pos={0 0}, upload=523833]: Uploading piece... Dec 30 13:05:18.237062679 (597h36m18.108145714s) [noise=true , address=145.79.227.34:4905 , piece_id=J32TZMXC5FM3FPJM674QCXWHRLN2KQT2F6KYNCM3KA5GO4UY4CHA, seg_pos={0 0}, upload=523833]: Uploading piece... Dec 30 13:05:18.237424669 (597h36m18.108507704s) [noise=true , address=77.90.31.21:4877 , piece_id=OVD3QYJFN6RPTEPXFSZWSXF3J2YXCXZTUEWWYJJMYF4OZ4BRCZDQ, seg_pos={0 0}, upload=523833]: Uploading piece... Dec 30 13:05:18.237720462 (597h36m18.108803497s) [noise=true , address=77.90.31.15:4697 , piece_id=KBZYZPT6HFCWY36ZQK37N6XQNZVFNOMLAGRAPJAA76MYQ2KBSX3A, seg_pos={0 0}, upload=523833]: Uploading piece... Dec 30 13:05:18.238346965 (597h36m18.10943s) [noise=true , address=145.79.227.46:4602 , piece_id=YG7Y3JZELTLE2NM2LLDLRXB3MU2TQKXZGPTKWM5VNRJIZHHRO27Q, seg_pos={0 0}, upload=523833]: Uploading piece... Dec 30 13:05:18.238818193 (597h36m18.109901228s) [noise=true , address=145.79.227.31:4799 , piece_id=X6UQZM7YVLMCR5X35EGQ7N6RBGYDYRJMI2YH7DV3CAORBDUMFHAQ, seg_pos={0 0}, upload=523833]: Uploading piece... Dec 30 13:05:18.239195892 (597h36m18.110278927s) [noise=true , address=145.79.227.12:4767 , piece_id=I2U5XKOLG2WSPVXF7MGXXSGJQIFFSJ46LZLNLATI7EMGYWTACI6Q, seg_pos={0 0}, upload=523833]: Uploading piece... Dec 30 13:05:18.239419397 (597h36m18.110502442s) [noise=true , address=145.79.227.20:4438 , piece_id=G2LUF234AWZ5GIL6DTUIWVBGTPR6B6RZDVGKBSRULCNZJOXDVN5Q, seg_pos={0 0}, upload=523833]: Uploading piece... Dec 30 13:05:19.938524132 (597h36m19.809607167s) [upload=523833]: Next returned nil segment Dec 30 13:05:19.938538439 (597h36m19.809621474s) [upload=523833]: Waiting for error group managing segments... Dec 30 13:05:20.225416789 (597h36m20.096499834s) [noise=true , address=145.79.227.46:4602 , piece_id=YG7Y3JZELTLE2NM2LLDLRXB3MU2TQKXZGPTKWM5VNRJIZHHRO27Q, seg_pos={0 0}, upload=523833]: Done uploading piece. err: <nil> Dec 30 13:05:20.230259981 (597h36m20.101343026s) [noise=true , address=145.79.227.3:4151 , piece_id=4SPC56GTTPAUOEQS6WIYKT4XS5I7TH4NJTNL2SH44CUB4AGC7S5Q, seg_pos={0 0}, upload=523833]: Done uploading piece. err: <nil> Dec 30 13:05:20.231961621 (597h36m20.103044666s) [noise=true , address=145.79.227.20:4438 , piece_id=G2LUF234AWZ5GIL6DTUIWVBGTPR6B6RZDVGKBSRULCNZJOXDVN5Q, seg_pos={0 0}, upload=523833]: Done uploading piece. err: <nil> Dec 30 13:05:20.236052922 (597h36m20.107135967s) [noise=true , address=145.79.227.52:4447 , piece_id=4UTPOVLC4FIAGMAGULQDRCKKS4FDSADZ36A62VHQ7Y5YJKWKSNZQ, seg_pos={0 0}, upload=523833]: Done uploading piece. err: <nil> Dec 30 13:05:20.245330678 (597h36m20.116413723s) [noise=true , address=145.79.227.9:4366 , piece_id=3HO7WFYJ5V7QMA4ULDRZ2QUDFC3X3ARDR2JLL4SBEH3R3442LEEQ, seg_pos={0 0}, upload=523833]: Done uploading piece. err: <nil> Dec 30 13:05:20.246275477 (597h36m20.117358512s) [noise=true , address=145.79.227.25:4617 , piece_id=3IPLSEQ4QUYBTD3J5RGBRVBB7MJ2BPJZ2SJWPLX2GULITVPHEZVQ, seg_pos={0 0}, upload=523833]: Done uploading piece. err: <nil> Dec 30 13:05:20.246765681 (597h36m20.117848726s) [noise=true , address=145.79.227.2:4115 , piece_id=Y3ZWB3BFPATJ3NUY7OUVPUO3KJ7QSURWXP6EGJY4D6CRLIYJSZ3Q, seg_pos={0 0}, upload=523833]: Done uploading piece. err: <nil> Dec 30 13:05:20.247080049 (597h36m20.118163084s) [noise=true , address=145.79.227.32:4834 , piece_id=6CVFHJDNVJXYCFQSKYJ4JGE45AFLLUZJBKG53GF7TGPOPUBG2YLQ, seg_pos={0 0}, upload=523833]: Done uploading piece. err: <nil> Dec 30 13:05:20.271725659 (597h36m20.142808704s) [noise=true , address=145.79.227.31:4799 , piece_id=X6UQZM7YVLMCR5X35EGQ7N6RBGYDYRJMI2YH7DV3CAORBDUMFHAQ, seg_pos={0 0}, upload=523833]: Done uploading piece. err: <nil> Dec 30 13:05:20.306214921 (597h36m20.177297956s) [noise=true , address=145.79.227.12:4767 , piece_id=I2U5XKOLG2WSPVXF7MGXXSGJQIFFSJ46LZLNLATI7EMGYWTACI6Q, seg_pos={0 0}, upload=523833]: Done uploading piece. err: <nil> Dec 30 13:05:20.311072280 (597h36m20.182155325s) [noise=true , address=145.79.227.50:4375 , piece_id=BYGKORNIBEJSJYBLZTWF2JHW4DC65I4O5WQG2AUU6AZBU2NVTEVA, seg_pos={0 0}, upload=523833]: Done uploading piece. err: <nil> Dec 30 13:05:20.394910160 (597h36m20.265993206s) [noise=true , address=77.90.31.15:4697 , piece_id=KBZYZPT6HFCWY36ZQK37N6XQNZVFNOMLAGRAPJAA76MYQ2KBSX3A, seg_pos={0 0}, upload=523833]: Done uploading piece. err: <nil> Dec 30 13:05:20.484099100 (597h36m20.355182136s) [noise=true , address=77.90.31.21:4877 , piece_id=OVD3QYJFN6RPTEPXFSZWSXF3J2YXCXZTUEWWYJJMYF4OZ4BRCZDQ, seg_pos={0 0}, upload=523833]: Done uploading piece. err: <nil> Dec 30 13:05:20.484177910 (597h36m20.355260965s) [seg_pos={0 0}, upload=523833]: Segment reached optimal threshold of 13 pieces. Dec 30 13:05:20.484451902 (597h36m20.355534947s) [noise=true , address=45.91.116.11:4117 , piece_id=CDYF2GSH5JXBJZ6U4MLTRZPKJV7JMZDOC7UXWRYLWVDLSYKRJK5Q, seg_pos={0 0}, upload=523833]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6): protocol: expected piece hash; context canceled; EOF Dec 30 13:05:20.484461801 (597h36m20.355544836s) [noise=true , address=45.91.116.11:4117 , piece_id=CDYF2GSH5JXBJZ6U4MLTRZPKJV7JMZDOC7UXWRYLWVDLSYKRJK5Q, seg_pos={0 0}, upload=523833]: Long tail cancellation due to optimal threshold Dec 30 13:05:20.484631704 (597h36m20.355714749s) [noise=true , address=145.79.227.34:4905 , piece_id=J32TZMXC5FM3FPJM674QCXWHRLN2KQT2F6KYNCM3KA5GO4UY4CHA, seg_pos={0 0}, upload=523833]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ffKESNSVpXQzbwJinP6vvgE6zoF3RFz2NRUuuJHvd7bvk1gc9): protocol: expected piece hash; context canceled; EOF Dec 30 13:05:20.484643737 (597h36m20.355726772s) [noise=true , address=145.79.227.34:4905 , piece_id=J32TZMXC5FM3FPJM674QCXWHRLN2KQT2F6KYNCM3KA5GO4UY4CHA, seg_pos={0 0}, upload=523833]: Long tail cancellation due to optimal threshold Dec 30 13:05:20.484847886 (597h36m20.355930931s) [noise=true , address=77.90.31.13:4626 , piece_id=V27YSE6V3NJYGPHWOXJSJUNXNWXFPUX3TEOK4B7VXSFIA26Y5CPQ, seg_pos={0 0}, upload=523833]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu): protocol: context canceled; context canceled Dec 30 13:05:20.484857505 (597h36m20.35594054s) [noise=true , address=77.90.31.13:4626 , piece_id=V27YSE6V3NJYGPHWOXJSJUNXNWXFPUX3TEOK4B7VXSFIA26Y5CPQ, seg_pos={0 0}, upload=523833]: Long tail cancellation due to optimal threshold Dec 30 13:05:20.484881309 (597h36m20.355964355s) [seg_pos={0 0}, upload=523833]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 13:05:20.484989906 (597h36m20.356072951s) [upload=523833]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 13:05:20.484997742 (597h36m20.356080777s) [upload=523833]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 13:05:20.779558830 (597h36m20.650641885s) [upload=523833]: Done uploading segments... Dec 30 13:05:20.779584489 (597h36m20.650667544s) [upload=523833]: Upload finished. err: <nil> Duration: 6.250143269s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 8 out of 60 is failed (threshold=3) [.......X.....X.X...X.........X....X........X..............X.] duration: 44.121570984s trace: https://jaeger.dev.storj.io/trace/59feccf4c7c3ba3a FAIL (2025-12-30 13:05:49 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: quiescence ======LOGS====== ec 30 13:04:06.775086282 (597h35m6.646169327s) [noise=true , address=145.79.227.40:4076 , piece_id=XGXIJP6AWHH322IXAUG3QEOO2AEPBEO6E52X6VGC226CAZUKQ7NA, seg_pos={0 0}, upload=523818]: Uploading piece... Dec 30 13:04:06.775516872 (597h35m6.646599907s) [noise=true , address=77.90.31.15:4708 , piece_id=K6SXMEM4SWGSWASJT46GOND2LF4REXOJ2RZJCPCGGQU7JJ2VGKDA, seg_pos={0 0}, upload=523818]: Uploading piece... Dec 30 13:04:06.775911253 (597h35m6.646994298s) [noise=true , address=64.49.13.27:4964 , piece_id=RFCAZJ226BWPK37KCUOIJ7UQ2UJLDQQ6YILNZO67FEONKRA5LB6A, seg_pos={0 0}, upload=523818]: Uploading piece... Dec 30 13:04:06.776366340 (597h35m6.647449395s) [noise=true , address=45.91.116.14:4222 , piece_id=4JWMPKJRZPU2FDGXCS5JW5CP6OL3T3XLNGEUTXEGSDBBKHMUFIOA, seg_pos={0 0}, upload=523818]: Uploading piece... Dec 30 13:04:06.777866967 (597h35m6.648950012s) [noise=true , address=145.79.227.34:4936 , piece_id=K3HBCI7M3WQIJ3GMWMQN33F3UQPWUMQW33NT5H52BNKDD7QNCQRQ, seg_pos={0 0}, upload=523818]: Uploading piece... Dec 30 13:04:06.778495334 (597h35m6.649578389s) [noise=true , address=145.79.227.5:4253 , piece_id=LOBQRRK5BLWDTAPUJB42YT3NC2G6M2J7EU3ALUMLTRD77NPYYFLQ, seg_pos={0 0}, upload=523818]: Uploading piece... Dec 30 13:04:06.779041454 (597h35m6.650124499s) [noise=true , address=77.90.31.18:4823 , piece_id=CZMTHJYNM5JF6VOFS7FTNABHNIWJTUTPHFRYYR63E2NNIKKCTWEQ, seg_pos={0 0}, upload=523818]: Uploading piece... Dec 30 13:04:06.779558509 (597h35m6.650641554s) [noise=true , address=45.91.116.8:4061 , piece_id=G5N5V7ISYQYRIPRSQI6YKO6HESVJOIF7N4GDYY7UJCPCUAZUER6A, seg_pos={0 0}, upload=523818]: Uploading piece... Dec 30 13:04:06.780053321 (597h35m6.651136366s) [noise=true , address=77.90.31.18:4836 , piece_id=NQ3OA633Y5TGCVGP5QCDHVG762ERUB5MB5GDLTCXIAX5S5XLJUQA, seg_pos={0 0}, upload=523818]: Uploading piece... Dec 30 13:04:06.780550918 (597h35m6.651633963s) [noise=true , address=77.90.31.12:4597 , piece_id=YMNNNTOZWOQ2VYT4VWVW7VHIUCQKZ6GMQMAYSUEQAEISXYO4SKRA, seg_pos={0 0}, upload=523818]: Uploading piece... Dec 30 13:04:06.781069085 (597h35m6.65215213s) [noise=true , address=145.79.227.51:4413 , piece_id=IGKBEDGRF5ZTI7UUPASHF5IFZWJSJZ2V6ATPHWTNW3SRMDSPRCIA, seg_pos={0 0}, upload=523818]: Uploading piece... Dec 30 13:04:06.781978246 (597h35m6.653061291s) [noise=true , address=145.79.227.49:4344 , piece_id=Z6LDJHQ6KLS3O6HDUIJI5EAGKH3Y7ZYIKFTXSGRVTTBOTVUCSRSQ, seg_pos={0 0}, upload=523818]: Uploading piece... Dec 30 13:04:06.782926150 (597h35m6.654009195s) [noise=true , address=145.79.227.44:4226 , piece_id=UPBKK6FMKBJTSNLDVHWP7BBVRMIHESW3ED7CQLUMEXRZGD7LYO5A, seg_pos={0 0}, upload=523818]: Uploading piece... Dec 30 13:04:06.783851333 (597h35m6.654934388s) [noise=true , address=45.91.116.9:4289 , piece_id=PFVRYB6YFVSYFD4ZSS7CDLORUXSG3VGG2SE4ZNE5YBFQTRL5MEJQ, seg_pos={0 0}, upload=523818]: Uploading piece... Dec 30 13:04:06.784805539 (597h35m6.655888584s) [noise=true , address=193.149.164.143:4703 , piece_id=WFBUJIEKQUXYFSSVB27M2YEEILDNN55E5KL23EUCUC2JDR3Z7XSQ, seg_pos={0 0}, upload=523818]: Uploading piece... Dec 30 13:04:06.786955854 (597h35m6.658038899s) [noise=true , address=145.79.227.30:4793 , piece_id=VWYFBUFM4NKIK7G7UXVUYAFZT3YYUW42W65PKWE4QKF7L2IO4YYA, seg_pos={0 0}, upload=523818]: Uploading piece... Dec 30 13:04:08.223121485 (597h35m8.09420453s) [upload=523818]: Next returned nil segment Dec 30 13:04:08.223139148 (597h35m8.094222193s) [upload=523818]: Waiting for error group managing segments... Dec 30 13:04:08.717719473 (597h35m8.588802518s) [noise=true , address=145.79.227.5:4253 , piece_id=LOBQRRK5BLWDTAPUJB42YT3NC2G6M2J7EU3ALUMLTRD77NPYYFLQ, seg_pos={0 0}, upload=523818]: Done uploading piece. err: <nil> Dec 30 13:04:08.717831707 (597h35m8.588914752s) [noise=true , address=145.79.227.51:4413 , piece_id=IGKBEDGRF5ZTI7UUPASHF5IFZWJSJZ2V6ATPHWTNW3SRMDSPRCIA, seg_pos={0 0}, upload=523818]: Done uploading piece. err: <nil> Dec 30 13:04:08.743536652 (597h35m8.614619687s) [noise=true , address=145.79.227.43:4187 , piece_id=NQCXR5HGRYEQ3WW27PQJLMUOEEUXZDGGQJQQO5NYGHPXPAQFWG7A, seg_pos={0 0}, upload=523818]: Done uploading piece. err: <nil> Dec 30 13:04:08.848591082 (597h35m8.719674127s) [noise=true , address=77.90.31.15:4708 , piece_id=K6SXMEM4SWGSWASJT46GOND2LF4REXOJ2RZJCPCGGQU7JJ2VGKDA, seg_pos={0 0}, upload=523818]: Done uploading piece. err: <nil> Dec 30 13:04:08.899414969 (597h35m8.770498014s) [noise=true , address=64.49.13.27:4964 , piece_id=RFCAZJ226BWPK37KCUOIJ7UQ2UJLDQQ6YILNZO67FEONKRA5LB6A, seg_pos={0 0}, upload=523818]: Done uploading piece. err: <nil> Dec 30 13:04:09.123129336 (597h35m8.994212371s) [noise=true , address=77.90.31.18:4836 , piece_id=NQ3OA633Y5TGCVGP5QCDHVG762ERUB5MB5GDLTCXIAX5S5XLJUQA, seg_pos={0 0}, upload=523818]: Done uploading piece. err: <nil> Dec 30 13:04:09.211969381 (597h35m9.083052416s) [noise=true , address=45.91.116.8:4061 , piece_id=G5N5V7ISYQYRIPRSQI6YKO6HESVJOIF7N4GDYY7UJCPCUAZUER6A, seg_pos={0 0}, upload=523818]: Done uploading piece. err: <nil> Dec 30 13:04:09.294556076 (597h35m9.165639142s) [noise=true , address=77.90.31.12:4597 , piece_id=YMNNNTOZWOQ2VYT4VWVW7VHIUCQKZ6GMQMAYSUEQAEISXYO4SKRA, seg_pos={0 0}, upload=523818]: Done uploading piece. err: <nil> Dec 30 13:04:09.479068837 (597h35m9.350151872s) [noise=true , address=193.149.164.143:4703 , piece_id=WFBUJIEKQUXYFSSVB27M2YEEILDNN55E5KL23EUCUC2JDR3Z7XSQ, seg_pos={0 0}, upload=523818]: Done uploading piece. err: <nil> Dec 30 13:04:09.510445983 (597h35m9.381529018s) [noise=true , address=145.79.227.44:4226 , piece_id=UPBKK6FMKBJTSNLDVHWP7BBVRMIHESW3ED7CQLUMEXRZGD7LYO5A, seg_pos={0 0}, upload=523818]: Done uploading piece. err: <nil> Dec 30 13:04:09.536218769 (597h35m9.407301814s) [noise=true , address=145.79.227.30:4793 , piece_id=VWYFBUFM4NKIK7G7UXVUYAFZT3YYUW42W65PKWE4QKF7L2IO4YYA, seg_pos={0 0}, upload=523818]: Done uploading piece. err: <nil> Dec 30 13:04:09.566859122 (597h35m9.437942167s) [noise=true , address=145.79.227.33:4877 , piece_id=BMMH4UJO7OQRDWK4QSSNCA3FVEUYXTX3TPWO62WMUG6DMQ5VTBEA, seg_pos={0 0}, upload=523818]: Done uploading piece. err: <nil> Dec 30 13:04:09.619658661 (597h35m9.490741696s) [noise=true , address=145.79.227.49:4344 , piece_id=Z6LDJHQ6KLS3O6HDUIJI5EAGKH3Y7ZYIKFTXSGRVTTBOTVUCSRSQ, seg_pos={0 0}, upload=523818]: Done uploading piece. err: <nil> Dec 30 13:04:09.621672355 (597h35m9.49275539s) [noise=true , address=45.91.116.14:4222 , piece_id=4JWMPKJRZPU2FDGXCS5JW5CP6OL3T3XLNGEUTXEGSDBBKHMUFIOA, seg_pos={0 0}, upload=523818]: Done uploading piece. err: <nil> Dec 30 13:04:09.621683898 (597h35m9.492766943s) [seg_pos={0 0}, upload=523818]: Segment reached optimal threshold of 14 pieces. Dec 30 13:04:09.621901051 (597h35m9.492984086s) [noise=true , address=77.90.31.18:4823 , piece_id=CZMTHJYNM5JF6VOFS7FTNABHNIWJTUTPHFRYYR63E2NNIKKCTWEQ, seg_pos={0 0}, upload=523818]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12uUCUDKZDwj25UcEKQ4WiTuQFMRbWw2oymGAdUdMX8wcUSd7jS): protocol: expected piece hash; context canceled; EOF Dec 30 13:04:09.621910259 (597h35m9.492993294s) [noise=true , address=77.90.31.18:4823 , piece_id=CZMTHJYNM5JF6VOFS7FTNABHNIWJTUTPHFRYYR63E2NNIKKCTWEQ, seg_pos={0 0}, upload=523818]: Long tail cancellation due to optimal threshold Dec 30 13:04:09.622063460 (597h35m9.493146505s) [noise=true , address=45.91.116.9:4289 , piece_id=PFVRYB6YFVSYFD4ZSS7CDLORUXSG3VGG2SE4ZNE5YBFQTRL5MEJQ, seg_pos={0 0}, upload=523818]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rfBAoVRdhYiRJFWqnamoRAFLxEUsk7tShJEG3jvhKDbcHwipD): protocol: write tcp 188.166.225.170:33424->45.91.116.9:4289: use of closed network connection; write tcp 188.166.225.170:33424->45.91.116.9:4289: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33424->45.91.116.9:4289: use of closed network connection Dec 30 13:04:09.622087205 (597h35m9.49317025s) [noise=true , address=45.91.116.9:4289 , piece_id=PFVRYB6YFVSYFD4ZSS7CDLORUXSG3VGG2SE4ZNE5YBFQTRL5MEJQ, seg_pos={0 0}, upload=523818]: Long tail cancellation due to optimal threshold Dec 30 13:04:09.622135457 (597h35m9.493218492s) [noise=true , address=145.79.227.40:4076 , piece_id=XGXIJP6AWHH322IXAUG3QEOO2AEPBEO6E52X6VGC226CAZUKQ7NA, seg_pos={0 0}, upload=523818]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19KofWeQV4jMQzmSCv3LUsUqAqGfAuDqyo9GGsqKGi7ctT3FA2): protocol: expected piece hash; context canceled; EOF Dec 30 13:04:09.622142100 (597h35m9.493225135s) [noise=true , address=145.79.227.40:4076 , piece_id=XGXIJP6AWHH322IXAUG3QEOO2AEPBEO6E52X6VGC226CAZUKQ7NA, seg_pos={0 0}, upload=523818]: Long tail cancellation due to optimal threshold Dec 30 13:04:09.622242572 (597h35m9.493325607s) [noise=true , address=145.79.227.34:4936 , piece_id=K3HBCI7M3WQIJ3GMWMQN33F3UQPWUMQW33NT5H52BNKDD7QNCQRQ, seg_pos={0 0}, upload=523818]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DicJh1hd5NUoj9uxoqzeaCmjGHwfk2VUaA9zQFmv72wBdBoW8): protocol: expected piece hash; context canceled; EOF Dec 30 13:04:09.622247882 (597h35m9.493330927s) [noise=true , address=145.79.227.34:4936 , piece_id=K3HBCI7M3WQIJ3GMWMQN33F3UQPWUMQW33NT5H52BNKDD7QNCQRQ, seg_pos={0 0}, upload=523818]: Long tail cancellation due to optimal threshold Dec 30 13:04:09.622337151 (597h35m9.493420186s) [seg_pos={0 0}, upload=523818]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 13:04:09.622456739 (597h35m9.493539795s) [upload=523818]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 13:04:09.622466388 (597h35m9.493549433s) [upload=523818]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 13:04:09.936998074 (597h35m9.808081119s) [upload=523818]: Done uploading segments... Dec 30 13:04:09.937018172 (597h35m9.808101217s) [upload=523818]: Upload finished. err: <nil> Duration: 44.12158937s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [X.......X.XX..X..X.XX...............X..X.X..X....X........XX] duration: 30.00290616s trace: https://jaeger.dev.storj.io/trace/4a02b772b0018a96 FAIL (2025-12-30 13:05:47 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, Piece ID: OVD3QYJFN6RPTEPXFSZWSXF3J2YXCXZTUEWWYJJMYF4OZ4BRCZDQ) context deadline exceeded; piecestore: (Node ID: 12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, Piece ID: KBZYZPT6HFCWY36ZQK37N6XQNZVFNOMLAGRAPJAA76MYQ2KBSX3A) context deadline exceeded ======LOGS====== Dec 30 13:05:17.988231417 (597h36m17.859314452s) [upload=523833]: Starting upload Dec 30 13:05:17.988290189 (597h36m17.859373224s) [upload=523833]: Done starting upload Dec 30 13:05:17.988476343 (597h36m17.859559378s) [upload=523833]: Uploading segments... Dec 30 13:05:17.988522001 (597h36m17.859605036s) [upload=523833]: Got next segment. Inline: false Dec 30 13:05:17.988548350 (597h36m17.859631385s) [upload=523833]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 13:05:17.988552870 (597h36m17.859635925s) [upload=523833]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 13:05:18.235197488 (597h36m18.106280523s) [seg_pos={0 0}, upload=523833]: Begin upload segment... Dec 30 13:05:18.235244077 (597h36m18.106327112s) [seg_pos={0 0}, upload=523833]: Done begin upload segment. Dec 30 13:05:18.235285987 (597h36m18.106369022s) [noise=true , address=145.79.227.9:4366 , piece_id=3HO7WFYJ5V7QMA4ULDRZ2QUDFC3X3ARDR2JLL4SBEH3R3442LEEQ, seg_pos={0 0}, upload=523833]: Uploading piece... Dec 30 13:05:18.235320202 (597h36m18.106403247s) [noise=true , address=45.91.116.11:4117 , piece_id=CDYF2GSH5JXBJZ6U4MLTRZPKJV7JMZDOC7UXWRYLWVDLSYKRJK5Q, seg_pos={0 0}, upload=523833]: Uploading piece... Dec 30 13:05:18.235847166 (597h36m18.106930201s) [noise=true , address=145.79.227.3:4151 , piece_id=4SPC56GTTPAUOEQS6WIYKT4XS5I7TH4NJTNL2SH44CUB4AGC7S5Q, seg_pos={0 0}, upload=523833]: Uploading piece... Dec 30 13:05:18.235886951 (597h36m18.106969996s) [noise=true , address=145.79.227.2:4115 , piece_id=Y3ZWB3BFPATJ3NUY7OUVPUO3KJ7QSURWXP6EGJY4D6CRLIYJSZ3Q, seg_pos={0 0}, upload=523833]: Uploading piece... Dec 30 13:05:18.236164309 (597h36m18.107247344s) [noise=true , address=77.90.31.13:4626 , piece_id=V27YSE6V3NJYGPHWOXJSJUNXNWXFPUX3TEOK4B7VXSFIA26Y5CPQ, seg_pos={0 0}, upload=523833]: Uploading piece... Dec 30 13:05:18.236275291 (597h36m18.107358336s) [noise=true , address=145.79.227.25:4617 , piece_id=3IPLSEQ4QUYBTD3J5RGBRVBB7MJ2BPJZ2SJWPLX2GULITVPHEZVQ, seg_pos={0 0}, upload=523833]: Uploading piece... Dec 30 13:05:18.236525807 (597h36m18.107608842s) [noise=true , address=145.79.227.32:4834 , piece_id=6CVFHJDNVJXYCFQSKYJ4JGE45AFLLUZJBKG53GF7TGPOPUBG2YLQ, seg_pos={0 0}, upload=523833]: Uploading piece... Dec 30 13:05:18.236699468 (597h36m18.107782503s) [noise=true , address=145.79.227.50:4375 , piece_id=BYGKORNIBEJSJYBLZTWF2JHW4DC65I4O5WQG2AUU6AZBU2NVTEVA, seg_pos={0 0}, upload=523833]: Uploading piece... Dec 30 13:05:18.236797896 (597h36m18.107880931s) [noise=true , address=145.79.227.52:4447 , piece_id=4UTPOVLC4FIAGMAGULQDRCKKS4FDSADZ36A62VHQ7Y5YJKWKSNZQ, seg_pos={0 0}, upload=523833]: Uploading piece... Dec 30 13:05:18.237062679 (597h36m18.108145714s) [noise=true , address=145.79.227.34:4905 , piece_id=J32TZMXC5FM3FPJM674QCXWHRLN2KQT2F6KYNCM3KA5GO4UY4CHA, seg_pos={0 0}, upload=523833]: Uploading piece... Dec 30 13:05:18.237424669 (597h36m18.108507704s) [noise=true , address=77.90.31.21:4877 , piece_id=OVD3QYJFN6RPTEPXFSZWSXF3J2YXCXZTUEWWYJJMYF4OZ4BRCZDQ, seg_pos={0 0}, upload=523833]: Uploading piece... Dec 30 13:05:18.237720462 (597h36m18.108803497s) [noise=true , address=77.90.31.15:4697 , piece_id=KBZYZPT6HFCWY36ZQK37N6XQNZVFNOMLAGRAPJAA76MYQ2KBSX3A, seg_pos={0 0}, upload=523833]: Uploading piece... Dec 30 13:05:18.238346965 (597h36m18.10943s) [noise=true , address=145.79.227.46:4602 , piece_id=YG7Y3JZELTLE2NM2LLDLRXB3MU2TQKXZGPTKWM5VNRJIZHHRO27Q, seg_pos={0 0}, upload=523833]: Uploading piece... Dec 30 13:05:18.238818193 (597h36m18.109901228s) [noise=true , address=145.79.227.31:4799 , piece_id=X6UQZM7YVLMCR5X35EGQ7N6RBGYDYRJMI2YH7DV3CAORBDUMFHAQ, seg_pos={0 0}, upload=523833]: Uploading piece... Dec 30 13:05:18.239195892 (597h36m18.110278927s) [noise=true , address=145.79.227.12:4767 , piece_id=I2U5XKOLG2WSPVXF7MGXXSGJQIFFSJ46LZLNLATI7EMGYWTACI6Q, seg_pos={0 0}, upload=523833]: Uploading piece... Dec 30 13:05:18.239419397 (597h36m18.110502442s) [noise=true , address=145.79.227.20:4438 , piece_id=G2LUF234AWZ5GIL6DTUIWVBGTPR6B6RZDVGKBSRULCNZJOXDVN5Q, seg_pos={0 0}, upload=523833]: Uploading piece... Dec 30 13:05:19.938524132 (597h36m19.809607167s) [upload=523833]: Next returned nil segment Dec 30 13:05:19.938538439 (597h36m19.809621474s) [upload=523833]: Waiting for error group managing segments... Dec 30 13:05:20.225416789 (597h36m20.096499834s) [noise=true , address=145.79.227.46:4602 , piece_id=YG7Y3JZELTLE2NM2LLDLRXB3MU2TQKXZGPTKWM5VNRJIZHHRO27Q, seg_pos={0 0}, upload=523833]: Done uploading piece. err: <nil> Dec 30 13:05:20.230259981 (597h36m20.101343026s) [noise=true , address=145.79.227.3:4151 , piece_id=4SPC56GTTPAUOEQS6WIYKT4XS5I7TH4NJTNL2SH44CUB4AGC7S5Q, seg_pos={0 0}, upload=523833]: Done uploading piece. err: <nil> Dec 30 13:05:20.231961621 (597h36m20.103044666s) [noise=true , address=145.79.227.20:4438 , piece_id=G2LUF234AWZ5GIL6DTUIWVBGTPR6B6RZDVGKBSRULCNZJOXDVN5Q, seg_pos={0 0}, upload=523833]: Done uploading piece. err: <nil> Dec 30 13:05:20.236052922 (597h36m20.107135967s) [noise=true , address=145.79.227.52:4447 , piece_id=4UTPOVLC4FIAGMAGULQDRCKKS4FDSADZ36A62VHQ7Y5YJKWKSNZQ, seg_pos={0 0}, upload=523833]: Done uploading piece. err: <nil> Dec 30 13:05:20.245330678 (597h36m20.116413723s) [noise=true , address=145.79.227.9:4366 , piece_id=3HO7WFYJ5V7QMA4ULDRZ2QUDFC3X3ARDR2JLL4SBEH3R3442LEEQ, seg_pos={0 0}, upload=523833]: Done uploading piece. err: <nil> Dec 30 13:05:20.246275477 (597h36m20.117358512s) [noise=true , address=145.79.227.25:4617 , piece_id=3IPLSEQ4QUYBTD3J5RGBRVBB7MJ2BPJZ2SJWPLX2GULITVPHEZVQ, seg_pos={0 0}, upload=523833]: Done uploading piece. err: <nil> Dec 30 13:05:20.246765681 (597h36m20.117848726s) [noise=true , address=145.79.227.2:4115 , piece_id=Y3ZWB3BFPATJ3NUY7OUVPUO3KJ7QSURWXP6EGJY4D6CRLIYJSZ3Q, seg_pos={0 0}, upload=523833]: Done uploading piece. err: <nil> Dec 30 13:05:20.247080049 (597h36m20.118163084s) [noise=true , address=145.79.227.32:4834 , piece_id=6CVFHJDNVJXYCFQSKYJ4JGE45AFLLUZJBKG53GF7TGPOPUBG2YLQ, seg_pos={0 0}, upload=523833]: Done uploading piece. err: <nil> Dec 30 13:05:20.271725659 (597h36m20.142808704s) [noise=true , address=145.79.227.31:4799 , piece_id=X6UQZM7YVLMCR5X35EGQ7N6RBGYDYRJMI2YH7DV3CAORBDUMFHAQ, seg_pos={0 0}, upload=523833]: Done uploading piece. err: <nil> Dec 30 13:05:20.306214921 (597h36m20.177297956s) [noise=true , address=145.79.227.12:4767 , piece_id=I2U5XKOLG2WSPVXF7MGXXSGJQIFFSJ46LZLNLATI7EMGYWTACI6Q, seg_pos={0 0}, upload=523833]: Done uploading piece. err: <nil> Dec 30 13:05:20.311072280 (597h36m20.182155325s) [noise=true , address=145.79.227.50:4375 , piece_id=BYGKORNIBEJSJYBLZTWF2JHW4DC65I4O5WQG2AUU6AZBU2NVTEVA, seg_pos={0 0}, upload=523833]: Done uploading piece. err: <nil> Dec 30 13:05:20.394910160 (597h36m20.265993206s) [noise=true , address=77.90.31.15:4697 , piece_id=KBZYZPT6HFCWY36ZQK37N6XQNZVFNOMLAGRAPJAA76MYQ2KBSX3A, seg_pos={0 0}, upload=523833]: Done uploading piece. err: <nil> Dec 30 13:05:20.484099100 (597h36m20.355182136s) [noise=true , address=77.90.31.21:4877 , piece_id=OVD3QYJFN6RPTEPXFSZWSXF3J2YXCXZTUEWWYJJMYF4OZ4BRCZDQ, seg_pos={0 0}, upload=523833]: Done uploading piece. err: <nil> Dec 30 13:05:20.484177910 (597h36m20.355260965s) [seg_pos={0 0}, upload=523833]: Segment reached optimal threshold of 13 pieces. Dec 30 13:05:20.484451902 (597h36m20.355534947s) [noise=true , address=45.91.116.11:4117 , piece_id=CDYF2GSH5JXBJZ6U4MLTRZPKJV7JMZDOC7UXWRYLWVDLSYKRJK5Q, seg_pos={0 0}, upload=523833]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6): protocol: expected piece hash; context canceled; EOF Dec 30 13:05:20.484461801 (597h36m20.355544836s) [noise=true , address=45.91.116.11:4117 , piece_id=CDYF2GSH5JXBJZ6U4MLTRZPKJV7JMZDOC7UXWRYLWVDLSYKRJK5Q, seg_pos={0 0}, upload=523833]: Long tail cancellation due to optimal threshold Dec 30 13:05:20.484631704 (597h36m20.355714749s) [noise=true , address=145.79.227.34:4905 , piece_id=J32TZMXC5FM3FPJM674QCXWHRLN2KQT2F6KYNCM3KA5GO4UY4CHA, seg_pos={0 0}, upload=523833]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ffKESNSVpXQzbwJinP6vvgE6zoF3RFz2NRUuuJHvd7bvk1gc9): protocol: expected piece hash; context canceled; EOF Dec 30 13:05:20.484643737 (597h36m20.355726772s) [noise=true , address=145.79.227.34:4905 , piece_id=J32TZMXC5FM3FPJM674QCXWHRLN2KQT2F6KYNCM3KA5GO4UY4CHA, seg_pos={0 0}, upload=523833]: Long tail cancellation due to optimal threshold Dec 30 13:05:20.484847886 (597h36m20.355930931s) [noise=true , address=77.90.31.13:4626 , piece_id=V27YSE6V3NJYGPHWOXJSJUNXNWXFPUX3TEOK4B7VXSFIA26Y5CPQ, seg_pos={0 0}, upload=523833]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu): protocol: context canceled; context canceled Dec 30 13:05:20.484857505 (597h36m20.35594054s) [noise=true , address=77.90.31.13:4626 , piece_id=V27YSE6V3NJYGPHWOXJSJUNXNWXFPUX3TEOK4B7VXSFIA26Y5CPQ, seg_pos={0 0}, upload=523833]: Long tail cancellation due to optimal threshold Dec 30 13:05:20.484881309 (597h36m20.355964355s) [seg_pos={0 0}, upload=523833]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 13:05:20.484989906 (597h36m20.356072951s) [upload=523833]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 13:05:20.484997742 (597h36m20.356080777s) [upload=523833]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 13:05:20.779558830 (597h36m20.650641885s) [upload=523833]: Done uploading segments... Dec 30 13:05:20.779584489 (597h36m20.650667544s) [upload=523833]: Upload finished. err: <nil> Duration: 30.002994069s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 47 out of 60 is failed (threshold=6) [.XXX...X.XXXXXXXXXXX.X.XXXXXXXXX.X.XXXXXX.X.XXX.XXXX.XXXXXXX] duration: 5.001384921s trace: https://jaeger.dev.storj.io/trace/2625bc641fd4fbc0 FAIL (2025-12-30 13:05:32 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001408016s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [...........................X........X...........X..X........] duration: 14.087024403s trace: https://jaeger.dev.storj.io/trace/17a3dfac1fc2c808 FAIL (2025-12-30 13:05:15 UTC) orbiter warning: context deadline exceeded Duration: 14.087057648s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [.............X.......................X....X.................] duration: 12.427716508s trace: https://jaeger.dev.storj.io/trace/7b47a373389cd7c3 FAIL (2025-12-30 13:05:15 UTC) orbiter warning: context deadline exceeded Duration: 12.4277443s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 47 out of 60 is failed (threshold=6) [X.XXX...X.XXXXXXXXXXX.X.XXXXXXXXX.X.XXXXXX.X.XXX.XXXX.XXXXXX] duration: 5.000701859s trace: https://jaeger.dev.storj.io/trace/10d0cafcbd8fb240 FAIL (2025-12-30 13:05:11 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000831576s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 8 out of 60 is failed (threshold=3) [........X.....X.X...X.........X....X........X..............X] duration: 59.28833581s trace: https://jaeger.dev.storj.io/trace/59feccf4c7c3ba3a FAIL (2025-12-30 13:05:05 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: quiescence ======LOGS====== ec 30 13:04:06.775086282 (597h35m6.646169327s) [noise=true , address=145.79.227.40:4076 , piece_id=XGXIJP6AWHH322IXAUG3QEOO2AEPBEO6E52X6VGC226CAZUKQ7NA, seg_pos={0 0}, upload=523818]: Uploading piece... Dec 30 13:04:06.775516872 (597h35m6.646599907s) [noise=true , address=77.90.31.15:4708 , piece_id=K6SXMEM4SWGSWASJT46GOND2LF4REXOJ2RZJCPCGGQU7JJ2VGKDA, seg_pos={0 0}, upload=523818]: Uploading piece... Dec 30 13:04:06.775911253 (597h35m6.646994298s) [noise=true , address=64.49.13.27:4964 , piece_id=RFCAZJ226BWPK37KCUOIJ7UQ2UJLDQQ6YILNZO67FEONKRA5LB6A, seg_pos={0 0}, upload=523818]: Uploading piece... Dec 30 13:04:06.776366340 (597h35m6.647449395s) [noise=true , address=45.91.116.14:4222 , piece_id=4JWMPKJRZPU2FDGXCS5JW5CP6OL3T3XLNGEUTXEGSDBBKHMUFIOA, seg_pos={0 0}, upload=523818]: Uploading piece... Dec 30 13:04:06.777866967 (597h35m6.648950012s) [noise=true , address=145.79.227.34:4936 , piece_id=K3HBCI7M3WQIJ3GMWMQN33F3UQPWUMQW33NT5H52BNKDD7QNCQRQ, seg_pos={0 0}, upload=523818]: Uploading piece... Dec 30 13:04:06.778495334 (597h35m6.649578389s) [noise=true , address=145.79.227.5:4253 , piece_id=LOBQRRK5BLWDTAPUJB42YT3NC2G6M2J7EU3ALUMLTRD77NPYYFLQ, seg_pos={0 0}, upload=523818]: Uploading piece... Dec 30 13:04:06.779041454 (597h35m6.650124499s) [noise=true , address=77.90.31.18:4823 , piece_id=CZMTHJYNM5JF6VOFS7FTNABHNIWJTUTPHFRYYR63E2NNIKKCTWEQ, seg_pos={0 0}, upload=523818]: Uploading piece... Dec 30 13:04:06.779558509 (597h35m6.650641554s) [noise=true , address=45.91.116.8:4061 , piece_id=G5N5V7ISYQYRIPRSQI6YKO6HESVJOIF7N4GDYY7UJCPCUAZUER6A, seg_pos={0 0}, upload=523818]: Uploading piece... Dec 30 13:04:06.780053321 (597h35m6.651136366s) [noise=true , address=77.90.31.18:4836 , piece_id=NQ3OA633Y5TGCVGP5QCDHVG762ERUB5MB5GDLTCXIAX5S5XLJUQA, seg_pos={0 0}, upload=523818]: Uploading piece... Dec 30 13:04:06.780550918 (597h35m6.651633963s) [noise=true , address=77.90.31.12:4597 , piece_id=YMNNNTOZWOQ2VYT4VWVW7VHIUCQKZ6GMQMAYSUEQAEISXYO4SKRA, seg_pos={0 0}, upload=523818]: Uploading piece... Dec 30 13:04:06.781069085 (597h35m6.65215213s) [noise=true , address=145.79.227.51:4413 , piece_id=IGKBEDGRF5ZTI7UUPASHF5IFZWJSJZ2V6ATPHWTNW3SRMDSPRCIA, seg_pos={0 0}, upload=523818]: Uploading piece... Dec 30 13:04:06.781978246 (597h35m6.653061291s) [noise=true , address=145.79.227.49:4344 , piece_id=Z6LDJHQ6KLS3O6HDUIJI5EAGKH3Y7ZYIKFTXSGRVTTBOTVUCSRSQ, seg_pos={0 0}, upload=523818]: Uploading piece... Dec 30 13:04:06.782926150 (597h35m6.654009195s) [noise=true , address=145.79.227.44:4226 , piece_id=UPBKK6FMKBJTSNLDVHWP7BBVRMIHESW3ED7CQLUMEXRZGD7LYO5A, seg_pos={0 0}, upload=523818]: Uploading piece... Dec 30 13:04:06.783851333 (597h35m6.654934388s) [noise=true , address=45.91.116.9:4289 , piece_id=PFVRYB6YFVSYFD4ZSS7CDLORUXSG3VGG2SE4ZNE5YBFQTRL5MEJQ, seg_pos={0 0}, upload=523818]: Uploading piece... Dec 30 13:04:06.784805539 (597h35m6.655888584s) [noise=true , address=193.149.164.143:4703 , piece_id=WFBUJIEKQUXYFSSVB27M2YEEILDNN55E5KL23EUCUC2JDR3Z7XSQ, seg_pos={0 0}, upload=523818]: Uploading piece... Dec 30 13:04:06.786955854 (597h35m6.658038899s) [noise=true , address=145.79.227.30:4793 , piece_id=VWYFBUFM4NKIK7G7UXVUYAFZT3YYUW42W65PKWE4QKF7L2IO4YYA, seg_pos={0 0}, upload=523818]: Uploading piece... Dec 30 13:04:08.223121485 (597h35m8.09420453s) [upload=523818]: Next returned nil segment Dec 30 13:04:08.223139148 (597h35m8.094222193s) [upload=523818]: Waiting for error group managing segments... Dec 30 13:04:08.717719473 (597h35m8.588802518s) [noise=true , address=145.79.227.5:4253 , piece_id=LOBQRRK5BLWDTAPUJB42YT3NC2G6M2J7EU3ALUMLTRD77NPYYFLQ, seg_pos={0 0}, upload=523818]: Done uploading piece. err: <nil> Dec 30 13:04:08.717831707 (597h35m8.588914752s) [noise=true , address=145.79.227.51:4413 , piece_id=IGKBEDGRF5ZTI7UUPASHF5IFZWJSJZ2V6ATPHWTNW3SRMDSPRCIA, seg_pos={0 0}, upload=523818]: Done uploading piece. err: <nil> Dec 30 13:04:08.743536652 (597h35m8.614619687s) [noise=true , address=145.79.227.43:4187 , piece_id=NQCXR5HGRYEQ3WW27PQJLMUOEEUXZDGGQJQQO5NYGHPXPAQFWG7A, seg_pos={0 0}, upload=523818]: Done uploading piece. err: <nil> Dec 30 13:04:08.848591082 (597h35m8.719674127s) [noise=true , address=77.90.31.15:4708 , piece_id=K6SXMEM4SWGSWASJT46GOND2LF4REXOJ2RZJCPCGGQU7JJ2VGKDA, seg_pos={0 0}, upload=523818]: Done uploading piece. err: <nil> Dec 30 13:04:08.899414969 (597h35m8.770498014s) [noise=true , address=64.49.13.27:4964 , piece_id=RFCAZJ226BWPK37KCUOIJ7UQ2UJLDQQ6YILNZO67FEONKRA5LB6A, seg_pos={0 0}, upload=523818]: Done uploading piece. err: <nil> Dec 30 13:04:09.123129336 (597h35m8.994212371s) [noise=true , address=77.90.31.18:4836 , piece_id=NQ3OA633Y5TGCVGP5QCDHVG762ERUB5MB5GDLTCXIAX5S5XLJUQA, seg_pos={0 0}, upload=523818]: Done uploading piece. err: <nil> Dec 30 13:04:09.211969381 (597h35m9.083052416s) [noise=true , address=45.91.116.8:4061 , piece_id=G5N5V7ISYQYRIPRSQI6YKO6HESVJOIF7N4GDYY7UJCPCUAZUER6A, seg_pos={0 0}, upload=523818]: Done uploading piece. err: <nil> Dec 30 13:04:09.294556076 (597h35m9.165639142s) [noise=true , address=77.90.31.12:4597 , piece_id=YMNNNTOZWOQ2VYT4VWVW7VHIUCQKZ6GMQMAYSUEQAEISXYO4SKRA, seg_pos={0 0}, upload=523818]: Done uploading piece. err: <nil> Dec 30 13:04:09.479068837 (597h35m9.350151872s) [noise=true , address=193.149.164.143:4703 , piece_id=WFBUJIEKQUXYFSSVB27M2YEEILDNN55E5KL23EUCUC2JDR3Z7XSQ, seg_pos={0 0}, upload=523818]: Done uploading piece. err: <nil> Dec 30 13:04:09.510445983 (597h35m9.381529018s) [noise=true , address=145.79.227.44:4226 , piece_id=UPBKK6FMKBJTSNLDVHWP7BBVRMIHESW3ED7CQLUMEXRZGD7LYO5A, seg_pos={0 0}, upload=523818]: Done uploading piece. err: <nil> Dec 30 13:04:09.536218769 (597h35m9.407301814s) [noise=true , address=145.79.227.30:4793 , piece_id=VWYFBUFM4NKIK7G7UXVUYAFZT3YYUW42W65PKWE4QKF7L2IO4YYA, seg_pos={0 0}, upload=523818]: Done uploading piece. err: <nil> Dec 30 13:04:09.566859122 (597h35m9.437942167s) [noise=true , address=145.79.227.33:4877 , piece_id=BMMH4UJO7OQRDWK4QSSNCA3FVEUYXTX3TPWO62WMUG6DMQ5VTBEA, seg_pos={0 0}, upload=523818]: Done uploading piece. err: <nil> Dec 30 13:04:09.619658661 (597h35m9.490741696s) [noise=true , address=145.79.227.49:4344 , piece_id=Z6LDJHQ6KLS3O6HDUIJI5EAGKH3Y7ZYIKFTXSGRVTTBOTVUCSRSQ, seg_pos={0 0}, upload=523818]: Done uploading piece. err: <nil> Dec 30 13:04:09.621672355 (597h35m9.49275539s) [noise=true , address=45.91.116.14:4222 , piece_id=4JWMPKJRZPU2FDGXCS5JW5CP6OL3T3XLNGEUTXEGSDBBKHMUFIOA, seg_pos={0 0}, upload=523818]: Done uploading piece. err: <nil> Dec 30 13:04:09.621683898 (597h35m9.492766943s) [seg_pos={0 0}, upload=523818]: Segment reached optimal threshold of 14 pieces. Dec 30 13:04:09.621901051 (597h35m9.492984086s) [noise=true , address=77.90.31.18:4823 , piece_id=CZMTHJYNM5JF6VOFS7FTNABHNIWJTUTPHFRYYR63E2NNIKKCTWEQ, seg_pos={0 0}, upload=523818]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12uUCUDKZDwj25UcEKQ4WiTuQFMRbWw2oymGAdUdMX8wcUSd7jS): protocol: expected piece hash; context canceled; EOF Dec 30 13:04:09.621910259 (597h35m9.492993294s) [noise=true , address=77.90.31.18:4823 , piece_id=CZMTHJYNM5JF6VOFS7FTNABHNIWJTUTPHFRYYR63E2NNIKKCTWEQ, seg_pos={0 0}, upload=523818]: Long tail cancellation due to optimal threshold Dec 30 13:04:09.622063460 (597h35m9.493146505s) [noise=true , address=45.91.116.9:4289 , piece_id=PFVRYB6YFVSYFD4ZSS7CDLORUXSG3VGG2SE4ZNE5YBFQTRL5MEJQ, seg_pos={0 0}, upload=523818]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rfBAoVRdhYiRJFWqnamoRAFLxEUsk7tShJEG3jvhKDbcHwipD): protocol: write tcp 188.166.225.170:33424->45.91.116.9:4289: use of closed network connection; write tcp 188.166.225.170:33424->45.91.116.9:4289: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33424->45.91.116.9:4289: use of closed network connection Dec 30 13:04:09.622087205 (597h35m9.49317025s) [noise=true , address=45.91.116.9:4289 , piece_id=PFVRYB6YFVSYFD4ZSS7CDLORUXSG3VGG2SE4ZNE5YBFQTRL5MEJQ, seg_pos={0 0}, upload=523818]: Long tail cancellation due to optimal threshold Dec 30 13:04:09.622135457 (597h35m9.493218492s) [noise=true , address=145.79.227.40:4076 , piece_id=XGXIJP6AWHH322IXAUG3QEOO2AEPBEO6E52X6VGC226CAZUKQ7NA, seg_pos={0 0}, upload=523818]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19KofWeQV4jMQzmSCv3LUsUqAqGfAuDqyo9GGsqKGi7ctT3FA2): protocol: expected piece hash; context canceled; EOF Dec 30 13:04:09.622142100 (597h35m9.493225135s) [noise=true , address=145.79.227.40:4076 , piece_id=XGXIJP6AWHH322IXAUG3QEOO2AEPBEO6E52X6VGC226CAZUKQ7NA, seg_pos={0 0}, upload=523818]: Long tail cancellation due to optimal threshold Dec 30 13:04:09.622242572 (597h35m9.493325607s) [noise=true , address=145.79.227.34:4936 , piece_id=K3HBCI7M3WQIJ3GMWMQN33F3UQPWUMQW33NT5H52BNKDD7QNCQRQ, seg_pos={0 0}, upload=523818]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DicJh1hd5NUoj9uxoqzeaCmjGHwfk2VUaA9zQFmv72wBdBoW8): protocol: expected piece hash; context canceled; EOF Dec 30 13:04:09.622247882 (597h35m9.493330927s) [noise=true , address=145.79.227.34:4936 , piece_id=K3HBCI7M3WQIJ3GMWMQN33F3UQPWUMQW33NT5H52BNKDD7QNCQRQ, seg_pos={0 0}, upload=523818]: Long tail cancellation due to optimal threshold Dec 30 13:04:09.622337151 (597h35m9.493420186s) [seg_pos={0 0}, upload=523818]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 13:04:09.622456739 (597h35m9.493539795s) [upload=523818]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 13:04:09.622466388 (597h35m9.493549433s) [upload=523818]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 13:04:09.936998074 (597h35m9.808081119s) [upload=523818]: Done uploading segments... Dec 30 13:04:09.937018172 (597h35m9.808101217s) [upload=523818]: Upload finished. err: <nil> Duration: 59.288350748s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 47 out of 60 is failed (threshold=6) [XX.XXX...X.XXXXXXXXXXX.X.XXXXXXXXX.X.XXXXXX.X.XXX.XXXX.XXXXX] duration: 5.000554017s trace: https://jaeger.dev.storj.io/trace/5c946e06ce8bd3bc FAIL (2025-12-30 13:04:54 UTC) orbiter warning: context deadline exceeded Duration: 5.000577112s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [............................X........X...........X..X.......] duration: 25.592467734s trace: https://jaeger.dev.storj.io/trace/17a3dfac1fc2c808 FAIL (2025-12-30 13:04:45 UTC) orbiter warning: context deadline exceeded Duration: 25.592495547s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [.X.......X.XX..X..X.XX...............X..X.X..X....X........X] duration: 30.002129434s trace: https://jaeger.dev.storj.io/trace/39a4d3531a3ebe88 FAIL (2025-12-30 13:04:38 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, Piece ID: XMJ7PPLTKMDJWC5FYJFGOEFLMFYQCFPZLVJFSNN2IPQEU6FLQDDA) context deadline exceeded; piecestore: (Node ID: 138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n, Piece ID: DNGZBI6RD7454OWRJXHONL7EDIXF7AYRQSJDPPSJ6JXHIFFAFQNA) context deadline exceeded ======LOGS====== Dec 30 13:04:08.937001439 (597h35m8.808084484s) [upload=523819]: Starting upload Dec 30 13:04:08.937076812 (597h35m8.808159848s) [upload=523819]: Done starting upload Dec 30 13:04:08.937116288 (597h35m8.808199334s) [upload=523819]: Uploading segments... Dec 30 13:04:08.937414115 (597h35m8.80849716s) [upload=523819]: Got next segment. Inline: false Dec 30 13:04:08.937464972 (597h35m8.808548017s) [upload=523819]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 13:04:08.937471214 (597h35m8.808554259s) [upload=523819]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 13:04:09.182318129 (597h35m9.053401184s) [seg_pos={0 0}, upload=523819]: Begin upload segment... Dec 30 13:04:09.182388203 (597h35m9.053471238s) [seg_pos={0 0}, upload=523819]: Done begin upload segment. Dec 30 13:04:09.182446183 (597h35m9.053529229s) [noise=true , address=77.90.31.14:4662 , piece_id=DNGZBI6RD7454OWRJXHONL7EDIXF7AYRQSJDPPSJ6JXHIFFAFQNA, seg_pos={0 0}, upload=523819]: Uploading piece... Dec 30 13:04:09.182491119 (597h35m9.053574154s) [noise=true , address=45.91.116.14:4190 , piece_id=LWOMIF27CNKGDXVCJFTX66S2HFTO6VRCEUIVLKCK6OOWEYJP3LZA, seg_pos={0 0}, upload=523819]: Uploading piece... Dec 30 13:04:09.183022290 (597h35m9.054105335s) [noise=true , address=45.91.116.13:4155 , piece_id=VGMQRTR74XTFUVTF5MZH624UBNTPMSFZJNPREDBFKJW6TDCBFYAQ, seg_pos={0 0}, upload=523819]: Uploading piece... Dec 30 13:04:09.183485913 (597h35m9.054568958s) [noise=true , address=145.79.227.21:4474 , piece_id=D5EKMXW4VIY6OBEPFODEZFIQP56ZXIGU64TVXS4KKWRDVAC6INKA, seg_pos={0 0}, upload=523819]: Uploading piece... Dec 30 13:04:09.183943194 (597h35m9.055026239s) [noise=true , address=77.90.31.10:4520 , piece_id=ADVZ2MPJRVALFW7X62H6QP6EBUCF53XKTOCIKCEMAT5XA2JRK6OA, seg_pos={0 0}, upload=523819]: Uploading piece... Dec 30 13:04:09.183950357 (597h35m9.055033392s) [noise=true , address=45.91.116.10:4082 , piece_id=XMJ7PPLTKMDJWC5FYJFGOEFLMFYQCFPZLVJFSNN2IPQEU6FLQDDA, seg_pos={0 0}, upload=523819]: Uploading piece... Dec 30 13:04:09.184272410 (597h35m9.055355445s) [noise=true , address=145.79.227.17:4971 , piece_id=BFYJ7WHYCIIHJY4YIGFICVCPLY7RP3X6JX3KUYVIFUNPNB56S5FA, seg_pos={0 0}, upload=523819]: Uploading piece... Dec 30 13:04:09.184378182 (597h35m9.055461227s) [noise=true , address=145.79.227.45:4229 , piece_id=JRYZJA4SYFX4RESZHZ333DJ6SQ4UHRSZRACLSNU27EFRU6MZ7F6Q, seg_pos={0 0}, upload=523819]: Uploading piece... Dec 30 13:04:09.184593071 (597h35m9.055676106s) [noise=true , address=145.79.227.50:4374 , piece_id=YQR3DJOGBZKXYM6DCEI3PBYT4PXHJVWEGYZXALWDO4OBJM3Y4Y2Q, seg_pos={0 0}, upload=523819]: Uploading piece... Dec 30 13:04:09.184870389 (597h35m9.055953444s) [noise=true , address=145.79.227.51:4411 , piece_id=2P5VD42YYLLNTRD4QRA23OJHMHCQFVZWDHY34RGUIYEKGTZM53IA, seg_pos={0 0}, upload=523819]: Uploading piece... Dec 30 13:04:09.184955231 (597h35m9.056038266s) [noise=true , address=145.79.227.20:4439 , piece_id=IW4GYAGHQ3A3UVKZ6ZD6LWTSS4W5VMIFPT2E3HJE4R7VRAAENZNQ, seg_pos={0 0}, upload=523819]: Uploading piece... Dec 30 13:04:09.185452397 (597h35m9.056535442s) [noise=true , address=145.79.227.23:4546 , piece_id=YT7JBFYUVBPS6EOYLVYYSEUH6F3QZAIWMBGLNDRF77IJZNYHOL6A, seg_pos={0 0}, upload=523819]: Uploading piece... Dec 30 13:04:09.185474941 (597h35m9.056557976s) [noise=true , address=145.79.227.14:4839 , piece_id=I3DO7TVL25ZYLCGCFK62WO5AF2PPTXZBX7VGWNIKVLTVRM2JLBQA, seg_pos={0 0}, upload=523819]: Uploading piece... Dec 30 13:04:09.185840166 (597h35m9.056923201s) [noise=true , address=45.91.116.12:4228 , piece_id=3HYGMCS3TFTP2WDBGKYKBYMF7O44YBBDSLQZ5ZSMQK47FTLKFKPQ, seg_pos={0 0}, upload=523819]: Uploading piece... Dec 30 13:04:09.185990513 (597h35m9.057073548s) [noise=true , address=145.79.227.41:4087 , piece_id=32IAWL36ZJ5EBRPHVYXZHQ7C2E2JZ7VEIBTPQS54GEU5EVYWFWYQ, seg_pos={0 0}, upload=523819]: Uploading piece... Dec 30 13:04:09.186158492 (597h35m9.057241527s) [noise=true , address=77.90.31.16:4736 , piece_id=5QJEFJ2WDJS4Z2GF55RMUTP634V5UAM6STDTY7IFHYK6GYJDDG7Q, seg_pos={0 0}, upload=523819]: Uploading piece... Dec 30 13:04:10.873976214 (597h35m10.745059259s) [upload=523819]: Next returned nil segment Dec 30 13:04:10.874035537 (597h35m10.745118582s) [upload=523819]: Waiting for error group managing segments... Dec 30 13:04:11.161568834 (597h35m11.032651879s) [noise=true , address=145.79.227.20:4439 , piece_id=IW4GYAGHQ3A3UVKZ6ZD6LWTSS4W5VMIFPT2E3HJE4R7VRAAENZNQ, seg_pos={0 0}, upload=523819]: Done uploading piece. err: <nil> Dec 30 13:04:11.162712692 (597h35m11.033795737s) [noise=true , address=145.79.227.41:4087 , piece_id=32IAWL36ZJ5EBRPHVYXZHQ7C2E2JZ7VEIBTPQS54GEU5EVYWFWYQ, seg_pos={0 0}, upload=523819]: Done uploading piece. err: <nil> Dec 30 13:04:11.169500207 (597h35m11.040583252s) [noise=true , address=145.79.227.50:4374 , piece_id=YQR3DJOGBZKXYM6DCEI3PBYT4PXHJVWEGYZXALWDO4OBJM3Y4Y2Q, seg_pos={0 0}, upload=523819]: Done uploading piece. err: <nil> Dec 30 13:04:11.170996917 (597h35m11.042079962s) [noise=true , address=145.79.227.14:4839 , piece_id=I3DO7TVL25ZYLCGCFK62WO5AF2PPTXZBX7VGWNIKVLTVRM2JLBQA, seg_pos={0 0}, upload=523819]: Done uploading piece. err: <nil> Dec 30 13:04:11.176547756 (597h35m11.047630801s) [noise=true , address=145.79.227.23:4546 , piece_id=YT7JBFYUVBPS6EOYLVYYSEUH6F3QZAIWMBGLNDRF77IJZNYHOL6A, seg_pos={0 0}, upload=523819]: Done uploading piece. err: <nil> Dec 30 13:04:11.189877989 (597h35m11.060961034s) [noise=true , address=145.79.227.45:4229 , piece_id=JRYZJA4SYFX4RESZHZ333DJ6SQ4UHRSZRACLSNU27EFRU6MZ7F6Q, seg_pos={0 0}, upload=523819]: Done uploading piece. err: <nil> Dec 30 13:04:11.358971550 (597h35m11.230054595s) [noise=true , address=145.79.227.17:4971 , piece_id=BFYJ7WHYCIIHJY4YIGFICVCPLY7RP3X6JX3KUYVIFUNPNB56S5FA, seg_pos={0 0}, upload=523819]: Done uploading piece. err: <nil> Dec 30 13:04:11.361643517 (597h35m11.232726562s) [noise=true , address=77.90.31.14:4662 , piece_id=DNGZBI6RD7454OWRJXHONL7EDIXF7AYRQSJDPPSJ6JXHIFFAFQNA, seg_pos={0 0}, upload=523819]: Done uploading piece. err: <nil> Dec 30 13:04:11.484522337 (597h35m11.355605382s) [noise=true , address=77.90.31.16:4736 , piece_id=5QJEFJ2WDJS4Z2GF55RMUTP634V5UAM6STDTY7IFHYK6GYJDDG7Q, seg_pos={0 0}, upload=523819]: Done uploading piece. err: <nil> Dec 30 13:04:11.869505251 (597h35m11.740588296s) [noise=true , address=45.91.116.10:4082 , piece_id=XMJ7PPLTKMDJWC5FYJFGOEFLMFYQCFPZLVJFSNN2IPQEU6FLQDDA, seg_pos={0 0}, upload=523819]: Done uploading piece. err: <nil> Dec 30 13:04:11.875402761 (597h35m11.746485836s) [noise=true , address=145.79.227.21:4474 , piece_id=D5EKMXW4VIY6OBEPFODEZFIQP56ZXIGU64TVXS4KKWRDVAC6INKA, seg_pos={0 0}, upload=523819]: Done uploading piece. err: <nil> Dec 30 13:04:12.114251856 (597h35m11.985334901s) [noise=true , address=145.79.227.51:4411 , piece_id=2P5VD42YYLLNTRD4QRA23OJHMHCQFVZWDHY34RGUIYEKGTZM53IA, seg_pos={0 0}, upload=523819]: Done uploading piece. err: <nil> Dec 30 13:04:12.194154654 (597h35m12.065237829s) [noise=true , address=77.90.31.10:4520 , piece_id=ADVZ2MPJRVALFW7X62H6QP6EBUCF53XKTOCIKCEMAT5XA2JRK6OA, seg_pos={0 0}, upload=523819]: Done uploading piece. err: <nil> Dec 30 13:04:12.194179201 (597h35m12.065262246s) [seg_pos={0 0}, upload=523819]: Segment reached optimal threshold of 13 pieces. Dec 30 13:04:12.194449436 (597h35m12.065532471s) [noise=true , address=45.91.116.14:4190 , piece_id=LWOMIF27CNKGDXVCJFTX66S2HFTO6VRCEUIVLKCK6OOWEYJP3LZA, seg_pos={0 0}, upload=523819]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG): protocol: expected piece hash; context canceled; EOF Dec 30 13:04:12.194458703 (597h35m12.065541748s) [noise=true , address=45.91.116.14:4190 , piece_id=LWOMIF27CNKGDXVCJFTX66S2HFTO6VRCEUIVLKCK6OOWEYJP3LZA, seg_pos={0 0}, upload=523819]: Long tail cancellation due to optimal threshold Dec 30 13:04:12.194478841 (597h35m12.065561896s) [noise=true , address=45.91.116.13:4155 , piece_id=VGMQRTR74XTFUVTF5MZH624UBNTPMSFZJNPREDBFKJW6TDCBFYAQ, seg_pos={0 0}, upload=523819]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf): protocol: expected piece hash; context canceled; EOF Dec 30 13:04:12.194487829 (597h35m12.065570894s) [noise=true , address=45.91.116.13:4155 , piece_id=VGMQRTR74XTFUVTF5MZH624UBNTPMSFZJNPREDBFKJW6TDCBFYAQ, seg_pos={0 0}, upload=523819]: Long tail cancellation due to optimal threshold Dec 30 13:04:12.194647282 (597h35m12.065730327s) [noise=true , address=45.91.116.12:4228 , piece_id=3HYGMCS3TFTP2WDBGKYKBYMF7O44YBBDSLQZ5ZSMQK47FTLKFKPQ, seg_pos={0 0}, upload=523819]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA): protocol: write tcp 188.166.225.170:59610->45.91.116.12:4228: use of closed network connection; write tcp 188.166.225.170:59610->45.91.116.12:4228: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59610->45.91.116.12:4228: use of closed network connection Dec 30 13:04:12.194658153 (597h35m12.065741198s) [noise=true , address=45.91.116.12:4228 , piece_id=3HYGMCS3TFTP2WDBGKYKBYMF7O44YBBDSLQZ5ZSMQK47FTLKFKPQ, seg_pos={0 0}, upload=523819]: Long tail cancellation due to optimal threshold Dec 30 13:04:12.194681888 (597h35m12.065764943s) [seg_pos={0 0}, upload=523819]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 13:04:12.194767722 (597h35m12.065850787s) [upload=523819]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 13:04:12.194773613 (597h35m12.065856658s) [upload=523819]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 13:04:12.490787186 (597h35m12.361870231s) [upload=523819]: Done uploading segments... Dec 30 13:04:12.490805792 (597h35m12.361888827s) [upload=523819]: Upload finished. err: <nil> Duration: 30.002147728s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [..............X.......................X....X................] duration: 19.580898249s trace: https://jaeger.dev.storj.io/trace/7b47a373389cd7c3 FAIL (2025-12-30 13:04:27 UTC) orbiter warning: context deadline exceeded Duration: 19.580919859s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 7 out of 60 is failed (threshold=3) [.........X.....X.X...X.........X....X........X..............] duration: 46.187954785s trace: https://jaeger.dev.storj.io/trace/1e6433ec4a141391 FAIL (2025-12-30 13:04:05 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== [noise=true , address=77.90.31.11:4583 , piece_id=JEMHBN5CJQI4GTDPRB5RZLBRDUWB46JL7JTORRZXYJTLFK7Z66PA, seg_pos={0 0}, upload=523642]: Uploading piece... Dec 30 12:51:35.352305080 (597h22m35.223388115s) [noise=true , address=145.79.227.44:4222 , piece_id=32QNUAYLXL4AQYBRI6LDICOWQIZMSU3OVN5SWVOCB6XIZY4FHRSA, seg_pos={0 0}, upload=523642]: Uploading piece... Dec 30 12:51:35.353083031 (597h22m35.224166066s) [noise=true , address=145.79.227.34:4925 , piece_id=G46JE4KCDC7FPDEUW2YDTSLI5X66MAMDCRRLDG7EBYF6ENLELLEQ, seg_pos={0 0}, upload=523642]: Uploading piece... Dec 30 12:51:35.353482140 (597h22m35.224565175s) [noise=true , address=145.79.227.33:4897 , piece_id=6SGUOB6IMRF2SPO6WHSKB6XLQENVS5TCJ6RMO6BYIMQ2PANMHK6A, seg_pos={0 0}, upload=523642]: Uploading piece... Dec 30 12:51:35.354315267 (597h22m35.225398302s) [noise=true , address=145.79.227.17:4991 , piece_id=GUPSOAIXZGPRJ7MVZ46464IRG7FBQEFR2RHYDT2GIPAMAKN36YHQ, seg_pos={0 0}, upload=523642]: Uploading piece... Dec 30 12:51:35.354723675 (597h22m35.22580671s) [noise=true , address=145.79.227.2:4119 , piece_id=55AC2GONNJPSGZJ3Y3CYFRLUVKRDMZGLY4HMAPDMEGMN6AJCYWOQ, seg_pos={0 0}, upload=523642]: Uploading piece... Dec 30 12:51:35.355352552 (597h22m35.226435587s) [noise=true , address=64.49.13.169:4716 , piece_id=EIRJU5OSYOF3D3AWCGKKSPFL36THXWUWJWIMBYHVEPI7VB6ATD3Q, seg_pos={0 0}, upload=523642]: Uploading piece... Dec 30 12:51:35.356361994 (597h22m35.227445029s) [noise=true , address=145.79.227.49:4367 , piece_id=6L53GGMES5PBPHPLUCPDUGYKKOQQ64VSZWAGX6EP755TJXT5RH6A, seg_pos={0 0}, upload=523642]: Uploading piece... Dec 30 12:51:35.356578216 (597h22m35.227661261s) [noise=true , address=77.90.31.14:4682 , piece_id=XACXXUOJEEGBXQEJFZYOZKQDX5N2V3LX5NCEQP64YIBZPHK23NQA, seg_pos={0 0}, upload=523642]: Uploading piece... Dec 30 12:51:35.357840370 (597h22m35.228923405s) [noise=true , address=145.79.227.46:4612 , piece_id=VG2ZGVJNHBN2GT72UXK6QA32HYGW6465MEOII6GYSOS7SZYLBBJA, seg_pos={0 0}, upload=523642]: Uploading piece... Dec 30 12:51:35.359039433 (597h22m35.230122468s) [noise=true , address=145.79.227.40:4072 , piece_id=ACK67AVKO5NK7IJKXGL2SLN4GGEKGW47WNTCZCPHIAKIVXYGFHVA, seg_pos={0 0}, upload=523642]: Uploading piece... Dec 30 12:51:35.359333493 (597h22m35.230416538s) [noise=true , address=77.90.31.18:4812 , piece_id=V347IHYDQTVIQ43JTHQJJW2BU3E4VSOSWABCPJPVX2JVCR6J6UNA, seg_pos={0 0}, upload=523642]: Uploading piece... Dec 30 12:51:35.360694154 (597h22m35.231777189s) [noise=true , address=45.91.116.9:4289 , piece_id=O735F2PWWAXGOP54RXURLZ64S7XC4XZZY4OH7RM4N2UOIAA3LJTA, seg_pos={0 0}, upload=523642]: Uploading piece... Dec 30 12:51:35.361764952 (597h22m35.232847987s) [noise=true , address=145.79.227.43:4160 , piece_id=746FBAMI37I2B2UMEKDGOVGNDHN5EXJJH6CMGSCTEEOAVLNDXVXA, seg_pos={0 0}, upload=523642]: Uploading piece... Dec 30 12:51:35.363365710 (597h22m35.234448755s) [noise=true , address=145.79.227.19:4099 , piece_id=DLLN7EUVR2D72XD7BL57AW5JPHVE76PUWF6LWXDEZQH6JZVH3RZQ, seg_pos={0 0}, upload=523642]: Uploading piece... Dec 30 12:51:35.363684648 (597h22m35.234767683s) [noise=true , address=145.79.227.30:4775 , piece_id=JADWMGGTCCSYQMINZV4CLHKEAWREFOS5S4RSY2MFBKC3ULFBD7FA, seg_pos={0 0}, upload=523642]: Uploading piece... Dec 30 12:51:36.749009517 (597h22m36.620092552s) [upload=523642]: Next returned nil segment Dec 30 12:51:36.749023393 (597h22m36.620106428s) [upload=523642]: Waiting for error group managing segments... Dec 30 12:51:37.208596330 (597h22m37.079679376s) [noise=true , address=145.79.227.17:4991 , piece_id=GUPSOAIXZGPRJ7MVZ46464IRG7FBQEFR2RHYDT2GIPAMAKN36YHQ, seg_pos={0 0}, upload=523642]: Done uploading piece. err: <nil> Dec 30 12:51:37.240881687 (597h22m37.111964722s) [noise=true , address=145.79.227.49:4367 , piece_id=6L53GGMES5PBPHPLUCPDUGYKKOQQ64VSZWAGX6EP755TJXT5RH6A, seg_pos={0 0}, upload=523642]: Done uploading piece. err: <nil> Dec 30 12:51:37.294293140 (597h22m37.165376175s) [noise=true , address=145.79.227.2:4119 , piece_id=55AC2GONNJPSGZJ3Y3CYFRLUVKRDMZGLY4HMAPDMEGMN6AJCYWOQ, seg_pos={0 0}, upload=523642]: Done uploading piece. err: <nil> Dec 30 12:51:37.302040142 (597h22m37.173123177s) [noise=true , address=145.79.227.43:4160 , piece_id=746FBAMI37I2B2UMEKDGOVGNDHN5EXJJH6CMGSCTEEOAVLNDXVXA, seg_pos={0 0}, upload=523642]: Done uploading piece. err: <nil> Dec 30 12:51:37.308919160 (597h22m37.180002196s) [noise=true , address=145.79.227.19:4099 , piece_id=DLLN7EUVR2D72XD7BL57AW5JPHVE76PUWF6LWXDEZQH6JZVH3RZQ, seg_pos={0 0}, upload=523642]: Done uploading piece. err: <nil> Dec 30 12:51:37.370286944 (597h22m37.24136999s) [noise=true , address=77.90.31.10:4545 , piece_id=YQV3IPXMIBSO5ZCISFUBGRGHE6FIXLKTLNYKPWR42VNBKNOGPG5Q, seg_pos={0 0}, upload=523642]: Done uploading piece. err: <nil> Dec 30 12:51:37.397564325 (597h22m37.26864737s) [noise=true , address=77.90.31.18:4812 , piece_id=V347IHYDQTVIQ43JTHQJJW2BU3E4VSOSWABCPJPVX2JVCR6J6UNA, seg_pos={0 0}, upload=523642]: Done uploading piece. err: <nil> Dec 30 12:51:37.481372759 (597h22m37.352455794s) [noise=true , address=77.90.31.14:4682 , piece_id=XACXXUOJEEGBXQEJFZYOZKQDX5N2V3LX5NCEQP64YIBZPHK23NQA, seg_pos={0 0}, upload=523642]: Done uploading piece. err: <nil> Dec 30 12:51:37.868609147 (597h22m37.739692192s) [noise=true , address=145.79.227.46:4612 , piece_id=VG2ZGVJNHBN2GT72UXK6QA32HYGW6465MEOII6GYSOS7SZYLBBJA, seg_pos={0 0}, upload=523642]: Done uploading piece. err: <nil> Dec 30 12:51:38.122711215 (597h22m37.993794251s) [noise=true , address=145.79.227.44:4222 , piece_id=32QNUAYLXL4AQYBRI6LDICOWQIZMSU3OVN5SWVOCB6XIZY4FHRSA, seg_pos={0 0}, upload=523642]: Done uploading piece. err: <nil> Dec 30 12:51:38.156332275 (597h22m38.02741531s) [noise=true , address=145.79.227.30:4775 , piece_id=JADWMGGTCCSYQMINZV4CLHKEAWREFOS5S4RSY2MFBKC3ULFBD7FA, seg_pos={0 0}, upload=523642]: Done uploading piece. err: <nil> Dec 30 12:51:38.210830938 (597h22m38.081913983s) [noise=true , address=145.79.227.34:4925 , piece_id=G46JE4KCDC7FPDEUW2YDTSLI5X66MAMDCRRLDG7EBYF6ENLELLEQ, seg_pos={0 0}, upload=523642]: Done uploading piece. err: <nil> Dec 30 12:51:38.318916249 (597h22m38.189999284s) [noise=true , address=145.79.227.33:4897 , piece_id=6SGUOB6IMRF2SPO6WHSKB6XLQENVS5TCJ6RMO6BYIMQ2PANMHK6A, seg_pos={0 0}, upload=523642]: Done uploading piece. err: <nil> Dec 30 12:51:38.528107081 (597h22m38.399190127s) [noise=true , address=193.149.164.143:4694 , piece_id=AOZVLCCSBEYZX3WES7JJVOA2BECEBPII3A33PG5KCSQY6LSZ6Y3A, seg_pos={0 0}, upload=523642]: Done uploading piece. err: <nil> Dec 30 12:51:38.528133793 (597h22m38.399216838s) [seg_pos={0 0}, upload=523642]: Segment reached optimal threshold of 14 pieces. Dec 30 12:51:38.528364652 (597h22m38.399447687s) [noise=true , address=45.91.116.9:4289 , piece_id=O735F2PWWAXGOP54RXURLZ64S7XC4XZZY4OH7RM4N2UOIAA3LJTA, seg_pos={0 0}, upload=523642]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rfBAoVRdhYiRJFWqnamoRAFLxEUsk7tShJEG3jvhKDbcHwipD): protocol: expected piece hash; context canceled; EOF Dec 30 12:51:38.528373429 (597h22m38.399456464s) [noise=true , address=45.91.116.9:4289 , piece_id=O735F2PWWAXGOP54RXURLZ64S7XC4XZZY4OH7RM4N2UOIAA3LJTA, seg_pos={0 0}, upload=523642]: Long tail cancellation due to optimal threshold Dec 30 12:51:38.528672969 (597h22m38.399756014s) [noise=true , address=145.79.227.40:4072 , piece_id=ACK67AVKO5NK7IJKXGL2SLN4GGEKGW47WNTCZCPHIAKIVXYGFHVA, seg_pos={0 0}, upload=523642]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FSQUJNBPY52p2Wkg48DWPN3bdciabNacXcgDEHSwLRrnPG9ne): protocol: expected piece hash; context canceled; EOF Dec 30 12:51:38.528682567 (597h22m38.399765602s) [noise=true , address=145.79.227.40:4072 , piece_id=ACK67AVKO5NK7IJKXGL2SLN4GGEKGW47WNTCZCPHIAKIVXYGFHVA, seg_pos={0 0}, upload=523642]: Long tail cancellation due to optimal threshold Dec 30 12:51:38.528839085 (597h22m38.39992212s) [noise=true , address=77.90.31.11:4583 , piece_id=JEMHBN5CJQI4GTDPRB5RZLBRDUWB46JL7JTORRZXYJTLFK7Z66PA, seg_pos={0 0}, upload=523642]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Ttdx6LHmuAt8JVC7arCDgWjm9eLadjeaEw17F12btoXuSiq9t): protocol: expected piece hash; context canceled; EOF Dec 30 12:51:38.528847421 (597h22m38.399930476s) [noise=true , address=77.90.31.11:4583 , piece_id=JEMHBN5CJQI4GTDPRB5RZLBRDUWB46JL7JTORRZXYJTLFK7Z66PA, seg_pos={0 0}, upload=523642]: Long tail cancellation due to optimal threshold Dec 30 12:51:38.529001655 (597h22m38.4000847s) [noise=true , address=64.49.13.169:4716 , piece_id=EIRJU5OSYOF3D3AWCGKKSPFL36THXWUWJWIMBYHVEPI7VB6ATD3Q, seg_pos={0 0}, upload=523642]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1w21xeA7emhjphgYw2Qyz3uuHgPezuivmVD9KmrKmNZk3TnaGN): protocol: write tcp 188.166.225.170:59670->64.49.13.169:4716: use of closed network connection; write tcp 188.166.225.170:59670->64.49.13.169:4716: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59670->64.49.13.169:4716: use of closed network connection Dec 30 12:51:38.529010362 (597h22m38.400093407s) [noise=true , address=64.49.13.169:4716 , piece_id=EIRJU5OSYOF3D3AWCGKKSPFL36THXWUWJWIMBYHVEPI7VB6ATD3Q, seg_pos={0 0}, upload=523642]: Long tail cancellation due to optimal threshold Dec 30 12:51:38.529029568 (597h22m38.400112603s) [seg_pos={0 0}, upload=523642]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 12:51:38.529107577 (597h22m38.400190612s) [upload=523642]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 12:51:38.529114079 (597h22m38.400197124s) [upload=523642]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 12:51:38.828356312 (597h22m38.699439387s) [upload=523642]: Done uploading segments... Dec 30 12:51:38.828379166 (597h22m38.699462231s) [upload=523642]: Upload finished. err: <nil> Duration: 46.187984802s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 46 out of 60 is failed (threshold=6) [.XX.XXX...X.XXXXXXXXXXX.X.XXXXXXXXX.X.XXXXXX.X.XXX.XXXX.XXXX] duration: 5.000746925s trace: https://jaeger.dev.storj.io/trace/56b5f7834a7beb24 FAIL (2025-12-30 13:03:47 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000769768s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.............................X........X...........X..X......] duration: 19.185363845s trace: https://jaeger.dev.storj.io/trace/17a3dfac1fc2c808 FAIL (2025-12-30 13:03:25 UTC) orbiter warning: context deadline exceeded Duration: 19.185389875s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [...............X.......................X....X...............] duration: 17.351804308s trace: https://jaeger.dev.storj.io/trace/7b47a373389cd7c3 FAIL (2025-12-30 13:03:22 UTC) orbiter warning: context deadline exceeded Duration: 17.35182166s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [................X.......................X....X..............] duration: 18.785914707s trace: https://jaeger.dev.storj.io/trace/7b47a373389cd7c3 FAIL (2025-12-30 13:03:05 UTC) orbiter warning: context deadline exceeded Duration: 18.785937039s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [..X.......X.XX..X..X.XX...............X..X.X..X....X........] duration: 6.048968608s trace: https://jaeger.dev.storj.io/trace/23ad8207da0148f9 FAIL (2025-12-30 13:03:02 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, Piece ID: TIHHT6FYFWUDFIVQZVQLE5IAL5Q2E3TKHDDIUBZSGFBQKQS4W5GA) context deadline exceeded; piecestore: (Node ID: 1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, Piece ID: J27M26BILJ4V23GMBQGK4OVKICA4TE3LDARCMDFPH7W6RWBGUYKQ) context deadline exceeded ======LOGS====== ments... Dec 30 12:55:52.163563927 (597h26m52.034646982s) [upload=523705]: Got next segment. Inline: false Dec 30 12:55:52.163612819 (597h26m52.034695864s) [upload=523705]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 12:55:52.163618821 (597h26m52.034701866s) [upload=523705]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 12:55:52.410969244 (597h26m52.282052279s) [seg_pos={0 0}, upload=523705]: Begin upload segment... Dec 30 12:55:52.411037163 (597h26m52.282120198s) [seg_pos={0 0}, upload=523705]: Done begin upload segment. Dec 30 12:55:52.411066920 (597h26m52.282149955s) [noise=true , address=145.79.227.14:4838 , piece_id=AZX5QHDBV7AXA4FIKQ343QISBB3LH24ZHORG53CKKKGGXDXC4QWQ, seg_pos={0 0}, upload=523705]: Uploading piece... Dec 30 12:55:52.411202477 (597h26m52.282285522s) [noise=true , address=45.91.116.6:4973 , piece_id=OT2DELPCMUI22A444ULTMOO4CYN4VD32E654SR44MDN2WQUATGDA, seg_pos={0 0}, upload=523705]: Uploading piece... Dec 30 12:55:52.411721015 (597h26m52.28280405s) [noise=true , address=77.90.31.13:4626 , piece_id=TIHHT6FYFWUDFIVQZVQLE5IAL5Q2E3TKHDDIUBZSGFBQKQS4W5GA, seg_pos={0 0}, upload=523705]: Uploading piece... Dec 30 12:55:52.412140103 (597h26m52.283223138s) [noise=true , address=45.91.116.8:4048 , piece_id=UXYGO5E3UBXUJKN442OB6CJWQYGW6Y5XUGABC467V2EIZP4KILTA, seg_pos={0 0}, upload=523705]: Uploading piece... Dec 30 12:55:52.412521419 (597h26m52.283604454s) [noise=true , address=145.79.227.50:4373 , piece_id=BA2SH47KSKE7F5Q4N5HXVNYDZ5FR453LMZHUDBWZZZKYKOIRZXEQ, seg_pos={0 0}, upload=523705]: Uploading piece... Dec 30 12:55:52.412888228 (597h26m52.283971263s) [noise=true , address=145.79.227.27:4691 , piece_id=AG67HXKZBGSF3JV75G2RC5OXLUSKINQFPOOS2WNSR3R6P5FNPGLQ, seg_pos={0 0}, upload=523705]: Uploading piece... Dec 30 12:55:52.413237172 (597h26m52.284320207s) [noise=true , address=145.79.227.24:4581 , piece_id=XNJDC2HJ6NIEXHNMP33STICUMDD5PQ2Y7UGMWU65ZPNH27R5PVDQ, seg_pos={0 0}, upload=523705]: Uploading piece... Dec 30 12:55:52.413653144 (597h26m52.284736179s) [noise=true , address=145.79.227.21:4475 , piece_id=42PVG5Q7APUS6QHI5JP6H2OLYIJSN5FXFSTDCKSNZVRKYPWVNI6A, seg_pos={0 0}, upload=523705]: Uploading piece... Dec 30 12:55:52.414092440 (597h26m52.285175475s) [noise=true , address=45.91.116.13:4153 , piece_id=J27M26BILJ4V23GMBQGK4OVKICA4TE3LDARCMDFPH7W6RWBGUYKQ, seg_pos={0 0}, upload=523705]: Uploading piece... Dec 30 12:55:52.415787007 (597h26m52.286870042s) [noise=true , address=45.91.116.11:4118 , piece_id=2JG6PCUGTT3K6YUMTWNZQB3OXTDZ7RMAFYHHIJMHLVFDY2E5K75A, seg_pos={0 0}, upload=523705]: Uploading piece... Dec 30 12:55:52.415936882 (597h26m52.287019917s) [noise=true , address=45.91.116.14:4189 , piece_id=X7CQTC447CXK6ITDRX4FZN2DNAWCPAUDO2UYH7O2EDBBF2OS35JA, seg_pos={0 0}, upload=523705]: Uploading piece... Dec 30 12:55:52.416357675 (597h26m52.28744072s) [noise=true , address=77.90.31.10:4519 , piece_id=7Q52TQ7IO2B5RCQESR6HYMYW4CZRO5ZPDJPXEAFEKGDPW7FSWE4Q, seg_pos={0 0}, upload=523705]: Uploading piece... Dec 30 12:55:52.416794676 (597h26m52.287877721s) [noise=true , address=45.91.116.9:4262 , piece_id=NCLXSMOIPWB37ABGU6FUUSNFW5MUZ5PQT4YGT4YGCJ2QXNQ6ML6Q, seg_pos={0 0}, upload=523705]: Uploading piece... Dec 30 12:55:52.417238962 (597h26m52.288321997s) [noise=true , address=145.79.227.47:4267 , piece_id=YPMQM5AXQF3LHHNPVIQWNBUHILPO7ORH7P2DUOKG5LMWMXBNVGGA, seg_pos={0 0}, upload=523705]: Uploading piece... Dec 30 12:55:52.417715961 (597h26m52.288798996s) [noise=true , address=45.91.116.7:4012 , piece_id=NNXDGUVP755B6VQ7L6LFLRYE3KN7KDEMKHEUL7TS3GYPSBXHERHA, seg_pos={0 0}, upload=523705]: Uploading piece... Dec 30 12:55:52.418153584 (597h26m52.289236619s) [noise=true , address=145.79.227.20:4438 , piece_id=LRYZYP6O6RNIIPK2VLTX3A2IL34LBGFFOIINSAMZ3TY5LUNWTCKQ, seg_pos={0 0}, upload=523705]: Uploading piece... Dec 30 12:55:54.018730327 (597h26m53.889813362s) [upload=523705]: Next returned nil segment Dec 30 12:55:54.018743952 (597h26m53.889826997s) [upload=523705]: Waiting for error group managing segments... Dec 30 12:55:54.399128062 (597h26m54.270211097s) [noise=true , address=145.79.227.24:4581 , piece_id=XNJDC2HJ6NIEXHNMP33STICUMDD5PQ2Y7UGMWU65ZPNH27R5PVDQ, seg_pos={0 0}, upload=523705]: Done uploading piece. err: <nil> Dec 30 12:55:54.399875084 (597h26m54.270958129s) [noise=true , address=145.79.227.14:4838 , piece_id=AZX5QHDBV7AXA4FIKQ343QISBB3LH24ZHORG53CKKKGGXDXC4QWQ, seg_pos={0 0}, upload=523705]: Done uploading piece. err: <nil> Dec 30 12:55:54.404232521 (597h26m54.275315556s) [noise=true , address=145.79.227.27:4691 , piece_id=AG67HXKZBGSF3JV75G2RC5OXLUSKINQFPOOS2WNSR3R6P5FNPGLQ, seg_pos={0 0}, upload=523705]: Done uploading piece. err: <nil> Dec 30 12:55:54.424052321 (597h26m54.295135356s) [noise=true , address=145.79.227.20:4438 , piece_id=LRYZYP6O6RNIIPK2VLTX3A2IL34LBGFFOIINSAMZ3TY5LUNWTCKQ, seg_pos={0 0}, upload=523705]: Done uploading piece. err: <nil> Dec 30 12:55:54.433077286 (597h26m54.304160321s) [noise=true , address=145.79.227.21:4475 , piece_id=42PVG5Q7APUS6QHI5JP6H2OLYIJSN5FXFSTDCKSNZVRKYPWVNI6A, seg_pos={0 0}, upload=523705]: Done uploading piece. err: <nil> Dec 30 12:55:54.437149530 (597h26m54.308232575s) [noise=true , address=145.79.227.47:4267 , piece_id=YPMQM5AXQF3LHHNPVIQWNBUHILPO7ORH7P2DUOKG5LMWMXBNVGGA, seg_pos={0 0}, upload=523705]: Done uploading piece. err: <nil> Dec 30 12:55:54.480458237 (597h26m54.351541282s) [noise=true , address=145.79.227.50:4373 , piece_id=BA2SH47KSKE7F5Q4N5HXVNYDZ5FR453LMZHUDBWZZZKYKOIRZXEQ, seg_pos={0 0}, upload=523705]: Done uploading piece. err: <nil> Dec 30 12:55:54.689140229 (597h26m54.560223274s) [noise=true , address=77.90.31.13:4626 , piece_id=TIHHT6FYFWUDFIVQZVQLE5IAL5Q2E3TKHDDIUBZSGFBQKQS4W5GA, seg_pos={0 0}, upload=523705]: Done uploading piece. err: <nil> Dec 30 12:55:54.711163926 (597h26m54.582246961s) [noise=true , address=45.91.116.13:4153 , piece_id=J27M26BILJ4V23GMBQGK4OVKICA4TE3LDARCMDFPH7W6RWBGUYKQ, seg_pos={0 0}, upload=523705]: Done uploading piece. err: <nil> Dec 30 12:55:54.730262152 (597h26m54.601345197s) [noise=true , address=45.91.116.11:4118 , piece_id=2JG6PCUGTT3K6YUMTWNZQB3OXTDZ7RMAFYHHIJMHLVFDY2E5K75A, seg_pos={0 0}, upload=523705]: Done uploading piece. err: <nil> Dec 30 12:55:54.877886751 (597h26m54.748969796s) [noise=true , address=77.90.31.10:4519 , piece_id=7Q52TQ7IO2B5RCQESR6HYMYW4CZRO5ZPDJPXEAFEKGDPW7FSWE4Q, seg_pos={0 0}, upload=523705]: Done uploading piece. err: <nil> Dec 30 12:55:54.889062301 (597h26m54.760145336s) [noise=true , address=45.91.116.8:4048 , piece_id=UXYGO5E3UBXUJKN442OB6CJWQYGW6Y5XUGABC467V2EIZP4KILTA, seg_pos={0 0}, upload=523705]: Done uploading piece. err: <nil> Dec 30 12:55:55.074468032 (597h26m54.945551077s) [noise=true , address=45.91.116.9:4262 , piece_id=NCLXSMOIPWB37ABGU6FUUSNFW5MUZ5PQT4YGT4YGCJ2QXNQ6ML6Q, seg_pos={0 0}, upload=523705]: Done uploading piece. err: <nil> Dec 30 12:55:55.074489473 (597h26m54.945572518s) [seg_pos={0 0}, upload=523705]: Segment reached optimal threshold of 13 pieces. Dec 30 12:55:55.074727766 (597h26m54.945810811s) [noise=true , address=45.91.116.14:4189 , piece_id=X7CQTC447CXK6ITDRX4FZN2DNAWCPAUDO2UYH7O2EDBBF2OS35JA, seg_pos={0 0}, upload=523705]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: write tcp 188.166.225.170:57284->45.91.116.14:4189: use of closed network connection; write tcp 188.166.225.170:57284->45.91.116.14:4189: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57284->45.91.116.14:4189: use of closed network connection Dec 30 12:55:55.074735361 (597h26m54.945818396s) [noise=true , address=45.91.116.14:4189 , piece_id=X7CQTC447CXK6ITDRX4FZN2DNAWCPAUDO2UYH7O2EDBBF2OS35JA, seg_pos={0 0}, upload=523705]: Long tail cancellation due to optimal threshold Dec 30 12:55:55.074903922 (597h26m54.945986967s) [noise=true , address=45.91.116.7:4012 , piece_id=NNXDGUVP755B6VQ7L6LFLRYE3KN7KDEMKHEUL7TS3GYPSBXHERHA, seg_pos={0 0}, upload=523705]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t): protocol: write tcp 188.166.225.170:49534->45.91.116.7:4012: use of closed network connection; write tcp 188.166.225.170:49534->45.91.116.7:4012: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49534->45.91.116.7:4012: use of closed network connection Dec 30 12:55:55.074929440 (597h26m54.946012485s) [noise=true , address=45.91.116.7:4012 , piece_id=NNXDGUVP755B6VQ7L6LFLRYE3KN7KDEMKHEUL7TS3GYPSBXHERHA, seg_pos={0 0}, upload=523705]: Long tail cancellation due to optimal threshold Dec 30 12:55:55.074969206 (597h26m54.946052241s) [noise=true , address=45.91.116.6:4973 , piece_id=OT2DELPCMUI22A444ULTMOO4CYN4VD32E654SR44MDN2WQUATGDA, seg_pos={0 0}, upload=523705]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV): protocol: expected piece hash; context canceled; EOF Dec 30 12:55:55.074975178 (597h26m54.946058213s) [noise=true , address=45.91.116.6:4973 , piece_id=OT2DELPCMUI22A444ULTMOO4CYN4VD32E654SR44MDN2WQUATGDA, seg_pos={0 0}, upload=523705]: Long tail cancellation due to optimal threshold Dec 30 12:55:55.075042837 (597h26m54.946125872s) [seg_pos={0 0}, upload=523705]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 12:55:55.075106016 (597h26m54.946189051s) [upload=523705]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 12:55:55.075110695 (597h26m54.94619373s) [upload=523705]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 12:55:55.365579192 (597h26m55.236662227s) [upload=523705]: Done uploading segments... Dec 30 12:55:55.365601594 (597h26m55.236684639s) [upload=523705]: Upload finished. err: <nil> Duration: 6.04901679s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 46 out of 60 is failed (threshold=6) [X.XX.XXX...X.XXXXXXXXXXX.X.XXXXXXXXX.X.XXXXXX.X.XXX.XXXX.XXX] duration: 5.000862647s trace: https://jaeger.dev.storj.io/trace/e2f717c77a67ee9 FAIL (2025-12-30 13:02:57 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000892374s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 7 out of 60 is failed (threshold=3) [..........X.....X.X...X.........X....X........X.............] duration: 16.655859063s trace: https://jaeger.dev.storj.io/trace/1e6433ec4a141391 FAIL (2025-12-30 13:02:50 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== [noise=true , address=77.90.31.11:4583 , piece_id=JEMHBN5CJQI4GTDPRB5RZLBRDUWB46JL7JTORRZXYJTLFK7Z66PA, seg_pos={0 0}, upload=523642]: Uploading piece... Dec 30 12:51:35.352305080 (597h22m35.223388115s) [noise=true , address=145.79.227.44:4222 , piece_id=32QNUAYLXL4AQYBRI6LDICOWQIZMSU3OVN5SWVOCB6XIZY4FHRSA, seg_pos={0 0}, upload=523642]: Uploading piece... Dec 30 12:51:35.353083031 (597h22m35.224166066s) [noise=true , address=145.79.227.34:4925 , piece_id=G46JE4KCDC7FPDEUW2YDTSLI5X66MAMDCRRLDG7EBYF6ENLELLEQ, seg_pos={0 0}, upload=523642]: Uploading piece... Dec 30 12:51:35.353482140 (597h22m35.224565175s) [noise=true , address=145.79.227.33:4897 , piece_id=6SGUOB6IMRF2SPO6WHSKB6XLQENVS5TCJ6RMO6BYIMQ2PANMHK6A, seg_pos={0 0}, upload=523642]: Uploading piece... Dec 30 12:51:35.354315267 (597h22m35.225398302s) [noise=true , address=145.79.227.17:4991 , piece_id=GUPSOAIXZGPRJ7MVZ46464IRG7FBQEFR2RHYDT2GIPAMAKN36YHQ, seg_pos={0 0}, upload=523642]: Uploading piece... Dec 30 12:51:35.354723675 (597h22m35.22580671s) [noise=true , address=145.79.227.2:4119 , piece_id=55AC2GONNJPSGZJ3Y3CYFRLUVKRDMZGLY4HMAPDMEGMN6AJCYWOQ, seg_pos={0 0}, upload=523642]: Uploading piece... Dec 30 12:51:35.355352552 (597h22m35.226435587s) [noise=true , address=64.49.13.169:4716 , piece_id=EIRJU5OSYOF3D3AWCGKKSPFL36THXWUWJWIMBYHVEPI7VB6ATD3Q, seg_pos={0 0}, upload=523642]: Uploading piece... Dec 30 12:51:35.356361994 (597h22m35.227445029s) [noise=true , address=145.79.227.49:4367 , piece_id=6L53GGMES5PBPHPLUCPDUGYKKOQQ64VSZWAGX6EP755TJXT5RH6A, seg_pos={0 0}, upload=523642]: Uploading piece... Dec 30 12:51:35.356578216 (597h22m35.227661261s) [noise=true , address=77.90.31.14:4682 , piece_id=XACXXUOJEEGBXQEJFZYOZKQDX5N2V3LX5NCEQP64YIBZPHK23NQA, seg_pos={0 0}, upload=523642]: Uploading piece... Dec 30 12:51:35.357840370 (597h22m35.228923405s) [noise=true , address=145.79.227.46:4612 , piece_id=VG2ZGVJNHBN2GT72UXK6QA32HYGW6465MEOII6GYSOS7SZYLBBJA, seg_pos={0 0}, upload=523642]: Uploading piece... Dec 30 12:51:35.359039433 (597h22m35.230122468s) [noise=true , address=145.79.227.40:4072 , piece_id=ACK67AVKO5NK7IJKXGL2SLN4GGEKGW47WNTCZCPHIAKIVXYGFHVA, seg_pos={0 0}, upload=523642]: Uploading piece... Dec 30 12:51:35.359333493 (597h22m35.230416538s) [noise=true , address=77.90.31.18:4812 , piece_id=V347IHYDQTVIQ43JTHQJJW2BU3E4VSOSWABCPJPVX2JVCR6J6UNA, seg_pos={0 0}, upload=523642]: Uploading piece... Dec 30 12:51:35.360694154 (597h22m35.231777189s) [noise=true , address=45.91.116.9:4289 , piece_id=O735F2PWWAXGOP54RXURLZ64S7XC4XZZY4OH7RM4N2UOIAA3LJTA, seg_pos={0 0}, upload=523642]: Uploading piece... Dec 30 12:51:35.361764952 (597h22m35.232847987s) [noise=true , address=145.79.227.43:4160 , piece_id=746FBAMI37I2B2UMEKDGOVGNDHN5EXJJH6CMGSCTEEOAVLNDXVXA, seg_pos={0 0}, upload=523642]: Uploading piece... Dec 30 12:51:35.363365710 (597h22m35.234448755s) [noise=true , address=145.79.227.19:4099 , piece_id=DLLN7EUVR2D72XD7BL57AW5JPHVE76PUWF6LWXDEZQH6JZVH3RZQ, seg_pos={0 0}, upload=523642]: Uploading piece... Dec 30 12:51:35.363684648 (597h22m35.234767683s) [noise=true , address=145.79.227.30:4775 , piece_id=JADWMGGTCCSYQMINZV4CLHKEAWREFOS5S4RSY2MFBKC3ULFBD7FA, seg_pos={0 0}, upload=523642]: Uploading piece... Dec 30 12:51:36.749009517 (597h22m36.620092552s) [upload=523642]: Next returned nil segment Dec 30 12:51:36.749023393 (597h22m36.620106428s) [upload=523642]: Waiting for error group managing segments... Dec 30 12:51:37.208596330 (597h22m37.079679376s) [noise=true , address=145.79.227.17:4991 , piece_id=GUPSOAIXZGPRJ7MVZ46464IRG7FBQEFR2RHYDT2GIPAMAKN36YHQ, seg_pos={0 0}, upload=523642]: Done uploading piece. err: <nil> Dec 30 12:51:37.240881687 (597h22m37.111964722s) [noise=true , address=145.79.227.49:4367 , piece_id=6L53GGMES5PBPHPLUCPDUGYKKOQQ64VSZWAGX6EP755TJXT5RH6A, seg_pos={0 0}, upload=523642]: Done uploading piece. err: <nil> Dec 30 12:51:37.294293140 (597h22m37.165376175s) [noise=true , address=145.79.227.2:4119 , piece_id=55AC2GONNJPSGZJ3Y3CYFRLUVKRDMZGLY4HMAPDMEGMN6AJCYWOQ, seg_pos={0 0}, upload=523642]: Done uploading piece. err: <nil> Dec 30 12:51:37.302040142 (597h22m37.173123177s) [noise=true , address=145.79.227.43:4160 , piece_id=746FBAMI37I2B2UMEKDGOVGNDHN5EXJJH6CMGSCTEEOAVLNDXVXA, seg_pos={0 0}, upload=523642]: Done uploading piece. err: <nil> Dec 30 12:51:37.308919160 (597h22m37.180002196s) [noise=true , address=145.79.227.19:4099 , piece_id=DLLN7EUVR2D72XD7BL57AW5JPHVE76PUWF6LWXDEZQH6JZVH3RZQ, seg_pos={0 0}, upload=523642]: Done uploading piece. err: <nil> Dec 30 12:51:37.370286944 (597h22m37.24136999s) [noise=true , address=77.90.31.10:4545 , piece_id=YQV3IPXMIBSO5ZCISFUBGRGHE6FIXLKTLNYKPWR42VNBKNOGPG5Q, seg_pos={0 0}, upload=523642]: Done uploading piece. err: <nil> Dec 30 12:51:37.397564325 (597h22m37.26864737s) [noise=true , address=77.90.31.18:4812 , piece_id=V347IHYDQTVIQ43JTHQJJW2BU3E4VSOSWABCPJPVX2JVCR6J6UNA, seg_pos={0 0}, upload=523642]: Done uploading piece. err: <nil> Dec 30 12:51:37.481372759 (597h22m37.352455794s) [noise=true , address=77.90.31.14:4682 , piece_id=XACXXUOJEEGBXQEJFZYOZKQDX5N2V3LX5NCEQP64YIBZPHK23NQA, seg_pos={0 0}, upload=523642]: Done uploading piece. err: <nil> Dec 30 12:51:37.868609147 (597h22m37.739692192s) [noise=true , address=145.79.227.46:4612 , piece_id=VG2ZGVJNHBN2GT72UXK6QA32HYGW6465MEOII6GYSOS7SZYLBBJA, seg_pos={0 0}, upload=523642]: Done uploading piece. err: <nil> Dec 30 12:51:38.122711215 (597h22m37.993794251s) [noise=true , address=145.79.227.44:4222 , piece_id=32QNUAYLXL4AQYBRI6LDICOWQIZMSU3OVN5SWVOCB6XIZY4FHRSA, seg_pos={0 0}, upload=523642]: Done uploading piece. err: <nil> Dec 30 12:51:38.156332275 (597h22m38.02741531s) [noise=true , address=145.79.227.30:4775 , piece_id=JADWMGGTCCSYQMINZV4CLHKEAWREFOS5S4RSY2MFBKC3ULFBD7FA, seg_pos={0 0}, upload=523642]: Done uploading piece. err: <nil> Dec 30 12:51:38.210830938 (597h22m38.081913983s) [noise=true , address=145.79.227.34:4925 , piece_id=G46JE4KCDC7FPDEUW2YDTSLI5X66MAMDCRRLDG7EBYF6ENLELLEQ, seg_pos={0 0}, upload=523642]: Done uploading piece. err: <nil> Dec 30 12:51:38.318916249 (597h22m38.189999284s) [noise=true , address=145.79.227.33:4897 , piece_id=6SGUOB6IMRF2SPO6WHSKB6XLQENVS5TCJ6RMO6BYIMQ2PANMHK6A, seg_pos={0 0}, upload=523642]: Done uploading piece. err: <nil> Dec 30 12:51:38.528107081 (597h22m38.399190127s) [noise=true , address=193.149.164.143:4694 , piece_id=AOZVLCCSBEYZX3WES7JJVOA2BECEBPII3A33PG5KCSQY6LSZ6Y3A, seg_pos={0 0}, upload=523642]: Done uploading piece. err: <nil> Dec 30 12:51:38.528133793 (597h22m38.399216838s) [seg_pos={0 0}, upload=523642]: Segment reached optimal threshold of 14 pieces. Dec 30 12:51:38.528364652 (597h22m38.399447687s) [noise=true , address=45.91.116.9:4289 , piece_id=O735F2PWWAXGOP54RXURLZ64S7XC4XZZY4OH7RM4N2UOIAA3LJTA, seg_pos={0 0}, upload=523642]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rfBAoVRdhYiRJFWqnamoRAFLxEUsk7tShJEG3jvhKDbcHwipD): protocol: expected piece hash; context canceled; EOF Dec 30 12:51:38.528373429 (597h22m38.399456464s) [noise=true , address=45.91.116.9:4289 , piece_id=O735F2PWWAXGOP54RXURLZ64S7XC4XZZY4OH7RM4N2UOIAA3LJTA, seg_pos={0 0}, upload=523642]: Long tail cancellation due to optimal threshold Dec 30 12:51:38.528672969 (597h22m38.399756014s) [noise=true , address=145.79.227.40:4072 , piece_id=ACK67AVKO5NK7IJKXGL2SLN4GGEKGW47WNTCZCPHIAKIVXYGFHVA, seg_pos={0 0}, upload=523642]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FSQUJNBPY52p2Wkg48DWPN3bdciabNacXcgDEHSwLRrnPG9ne): protocol: expected piece hash; context canceled; EOF Dec 30 12:51:38.528682567 (597h22m38.399765602s) [noise=true , address=145.79.227.40:4072 , piece_id=ACK67AVKO5NK7IJKXGL2SLN4GGEKGW47WNTCZCPHIAKIVXYGFHVA, seg_pos={0 0}, upload=523642]: Long tail cancellation due to optimal threshold Dec 30 12:51:38.528839085 (597h22m38.39992212s) [noise=true , address=77.90.31.11:4583 , piece_id=JEMHBN5CJQI4GTDPRB5RZLBRDUWB46JL7JTORRZXYJTLFK7Z66PA, seg_pos={0 0}, upload=523642]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Ttdx6LHmuAt8JVC7arCDgWjm9eLadjeaEw17F12btoXuSiq9t): protocol: expected piece hash; context canceled; EOF Dec 30 12:51:38.528847421 (597h22m38.399930476s) [noise=true , address=77.90.31.11:4583 , piece_id=JEMHBN5CJQI4GTDPRB5RZLBRDUWB46JL7JTORRZXYJTLFK7Z66PA, seg_pos={0 0}, upload=523642]: Long tail cancellation due to optimal threshold Dec 30 12:51:38.529001655 (597h22m38.4000847s) [noise=true , address=64.49.13.169:4716 , piece_id=EIRJU5OSYOF3D3AWCGKKSPFL36THXWUWJWIMBYHVEPI7VB6ATD3Q, seg_pos={0 0}, upload=523642]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1w21xeA7emhjphgYw2Qyz3uuHgPezuivmVD9KmrKmNZk3TnaGN): protocol: write tcp 188.166.225.170:59670->64.49.13.169:4716: use of closed network connection; write tcp 188.166.225.170:59670->64.49.13.169:4716: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59670->64.49.13.169:4716: use of closed network connection Dec 30 12:51:38.529010362 (597h22m38.400093407s) [noise=true , address=64.49.13.169:4716 , piece_id=EIRJU5OSYOF3D3AWCGKKSPFL36THXWUWJWIMBYHVEPI7VB6ATD3Q, seg_pos={0 0}, upload=523642]: Long tail cancellation due to optimal threshold Dec 30 12:51:38.529029568 (597h22m38.400112603s) [seg_pos={0 0}, upload=523642]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 12:51:38.529107577 (597h22m38.400190612s) [upload=523642]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 12:51:38.529114079 (597h22m38.400197124s) [upload=523642]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 12:51:38.828356312 (597h22m38.699439387s) [upload=523642]: Done uploading segments... Dec 30 12:51:38.828379166 (597h22m38.699462231s) [upload=523642]: Upload finished. err: <nil> Duration: 16.655875184s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 7 out of 60 is failed (threshold=3) [...........X.....X.X...X.........X....X........X............] duration: 43.320740533s trace: https://jaeger.dev.storj.io/trace/1e6433ec4a141391 FAIL (2025-12-30 13:02:33 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== [noise=true , address=77.90.31.11:4583 , piece_id=JEMHBN5CJQI4GTDPRB5RZLBRDUWB46JL7JTORRZXYJTLFK7Z66PA, seg_pos={0 0}, upload=523642]: Uploading piece... Dec 30 12:51:35.352305080 (597h22m35.223388115s) [noise=true , address=145.79.227.44:4222 , piece_id=32QNUAYLXL4AQYBRI6LDICOWQIZMSU3OVN5SWVOCB6XIZY4FHRSA, seg_pos={0 0}, upload=523642]: Uploading piece... Dec 30 12:51:35.353083031 (597h22m35.224166066s) [noise=true , address=145.79.227.34:4925 , piece_id=G46JE4KCDC7FPDEUW2YDTSLI5X66MAMDCRRLDG7EBYF6ENLELLEQ, seg_pos={0 0}, upload=523642]: Uploading piece... Dec 30 12:51:35.353482140 (597h22m35.224565175s) [noise=true , address=145.79.227.33:4897 , piece_id=6SGUOB6IMRF2SPO6WHSKB6XLQENVS5TCJ6RMO6BYIMQ2PANMHK6A, seg_pos={0 0}, upload=523642]: Uploading piece... Dec 30 12:51:35.354315267 (597h22m35.225398302s) [noise=true , address=145.79.227.17:4991 , piece_id=GUPSOAIXZGPRJ7MVZ46464IRG7FBQEFR2RHYDT2GIPAMAKN36YHQ, seg_pos={0 0}, upload=523642]: Uploading piece... Dec 30 12:51:35.354723675 (597h22m35.22580671s) [noise=true , address=145.79.227.2:4119 , piece_id=55AC2GONNJPSGZJ3Y3CYFRLUVKRDMZGLY4HMAPDMEGMN6AJCYWOQ, seg_pos={0 0}, upload=523642]: Uploading piece... Dec 30 12:51:35.355352552 (597h22m35.226435587s) [noise=true , address=64.49.13.169:4716 , piece_id=EIRJU5OSYOF3D3AWCGKKSPFL36THXWUWJWIMBYHVEPI7VB6ATD3Q, seg_pos={0 0}, upload=523642]: Uploading piece... Dec 30 12:51:35.356361994 (597h22m35.227445029s) [noise=true , address=145.79.227.49:4367 , piece_id=6L53GGMES5PBPHPLUCPDUGYKKOQQ64VSZWAGX6EP755TJXT5RH6A, seg_pos={0 0}, upload=523642]: Uploading piece... Dec 30 12:51:35.356578216 (597h22m35.227661261s) [noise=true , address=77.90.31.14:4682 , piece_id=XACXXUOJEEGBXQEJFZYOZKQDX5N2V3LX5NCEQP64YIBZPHK23NQA, seg_pos={0 0}, upload=523642]: Uploading piece... Dec 30 12:51:35.357840370 (597h22m35.228923405s) [noise=true , address=145.79.227.46:4612 , piece_id=VG2ZGVJNHBN2GT72UXK6QA32HYGW6465MEOII6GYSOS7SZYLBBJA, seg_pos={0 0}, upload=523642]: Uploading piece... Dec 30 12:51:35.359039433 (597h22m35.230122468s) [noise=true , address=145.79.227.40:4072 , piece_id=ACK67AVKO5NK7IJKXGL2SLN4GGEKGW47WNTCZCPHIAKIVXYGFHVA, seg_pos={0 0}, upload=523642]: Uploading piece... Dec 30 12:51:35.359333493 (597h22m35.230416538s) [noise=true , address=77.90.31.18:4812 , piece_id=V347IHYDQTVIQ43JTHQJJW2BU3E4VSOSWABCPJPVX2JVCR6J6UNA, seg_pos={0 0}, upload=523642]: Uploading piece... Dec 30 12:51:35.360694154 (597h22m35.231777189s) [noise=true , address=45.91.116.9:4289 , piece_id=O735F2PWWAXGOP54RXURLZ64S7XC4XZZY4OH7RM4N2UOIAA3LJTA, seg_pos={0 0}, upload=523642]: Uploading piece... Dec 30 12:51:35.361764952 (597h22m35.232847987s) [noise=true , address=145.79.227.43:4160 , piece_id=746FBAMI37I2B2UMEKDGOVGNDHN5EXJJH6CMGSCTEEOAVLNDXVXA, seg_pos={0 0}, upload=523642]: Uploading piece... Dec 30 12:51:35.363365710 (597h22m35.234448755s) [noise=true , address=145.79.227.19:4099 , piece_id=DLLN7EUVR2D72XD7BL57AW5JPHVE76PUWF6LWXDEZQH6JZVH3RZQ, seg_pos={0 0}, upload=523642]: Uploading piece... Dec 30 12:51:35.363684648 (597h22m35.234767683s) [noise=true , address=145.79.227.30:4775 , piece_id=JADWMGGTCCSYQMINZV4CLHKEAWREFOS5S4RSY2MFBKC3ULFBD7FA, seg_pos={0 0}, upload=523642]: Uploading piece... Dec 30 12:51:36.749009517 (597h22m36.620092552s) [upload=523642]: Next returned nil segment Dec 30 12:51:36.749023393 (597h22m36.620106428s) [upload=523642]: Waiting for error group managing segments... Dec 30 12:51:37.208596330 (597h22m37.079679376s) [noise=true , address=145.79.227.17:4991 , piece_id=GUPSOAIXZGPRJ7MVZ46464IRG7FBQEFR2RHYDT2GIPAMAKN36YHQ, seg_pos={0 0}, upload=523642]: Done uploading piece. err: <nil> Dec 30 12:51:37.240881687 (597h22m37.111964722s) [noise=true , address=145.79.227.49:4367 , piece_id=6L53GGMES5PBPHPLUCPDUGYKKOQQ64VSZWAGX6EP755TJXT5RH6A, seg_pos={0 0}, upload=523642]: Done uploading piece. err: <nil> Dec 30 12:51:37.294293140 (597h22m37.165376175s) [noise=true , address=145.79.227.2:4119 , piece_id=55AC2GONNJPSGZJ3Y3CYFRLUVKRDMZGLY4HMAPDMEGMN6AJCYWOQ, seg_pos={0 0}, upload=523642]: Done uploading piece. err: <nil> Dec 30 12:51:37.302040142 (597h22m37.173123177s) [noise=true , address=145.79.227.43:4160 , piece_id=746FBAMI37I2B2UMEKDGOVGNDHN5EXJJH6CMGSCTEEOAVLNDXVXA, seg_pos={0 0}, upload=523642]: Done uploading piece. err: <nil> Dec 30 12:51:37.308919160 (597h22m37.180002196s) [noise=true , address=145.79.227.19:4099 , piece_id=DLLN7EUVR2D72XD7BL57AW5JPHVE76PUWF6LWXDEZQH6JZVH3RZQ, seg_pos={0 0}, upload=523642]: Done uploading piece. err: <nil> Dec 30 12:51:37.370286944 (597h22m37.24136999s) [noise=true , address=77.90.31.10:4545 , piece_id=YQV3IPXMIBSO5ZCISFUBGRGHE6FIXLKTLNYKPWR42VNBKNOGPG5Q, seg_pos={0 0}, upload=523642]: Done uploading piece. err: <nil> Dec 30 12:51:37.397564325 (597h22m37.26864737s) [noise=true , address=77.90.31.18:4812 , piece_id=V347IHYDQTVIQ43JTHQJJW2BU3E4VSOSWABCPJPVX2JVCR6J6UNA, seg_pos={0 0}, upload=523642]: Done uploading piece. err: <nil> Dec 30 12:51:37.481372759 (597h22m37.352455794s) [noise=true , address=77.90.31.14:4682 , piece_id=XACXXUOJEEGBXQEJFZYOZKQDX5N2V3LX5NCEQP64YIBZPHK23NQA, seg_pos={0 0}, upload=523642]: Done uploading piece. err: <nil> Dec 30 12:51:37.868609147 (597h22m37.739692192s) [noise=true , address=145.79.227.46:4612 , piece_id=VG2ZGVJNHBN2GT72UXK6QA32HYGW6465MEOII6GYSOS7SZYLBBJA, seg_pos={0 0}, upload=523642]: Done uploading piece. err: <nil> Dec 30 12:51:38.122711215 (597h22m37.993794251s) [noise=true , address=145.79.227.44:4222 , piece_id=32QNUAYLXL4AQYBRI6LDICOWQIZMSU3OVN5SWVOCB6XIZY4FHRSA, seg_pos={0 0}, upload=523642]: Done uploading piece. err: <nil> Dec 30 12:51:38.156332275 (597h22m38.02741531s) [noise=true , address=145.79.227.30:4775 , piece_id=JADWMGGTCCSYQMINZV4CLHKEAWREFOS5S4RSY2MFBKC3ULFBD7FA, seg_pos={0 0}, upload=523642]: Done uploading piece. err: <nil> Dec 30 12:51:38.210830938 (597h22m38.081913983s) [noise=true , address=145.79.227.34:4925 , piece_id=G46JE4KCDC7FPDEUW2YDTSLI5X66MAMDCRRLDG7EBYF6ENLELLEQ, seg_pos={0 0}, upload=523642]: Done uploading piece. err: <nil> Dec 30 12:51:38.318916249 (597h22m38.189999284s) [noise=true , address=145.79.227.33:4897 , piece_id=6SGUOB6IMRF2SPO6WHSKB6XLQENVS5TCJ6RMO6BYIMQ2PANMHK6A, seg_pos={0 0}, upload=523642]: Done uploading piece. err: <nil> Dec 30 12:51:38.528107081 (597h22m38.399190127s) [noise=true , address=193.149.164.143:4694 , piece_id=AOZVLCCSBEYZX3WES7JJVOA2BECEBPII3A33PG5KCSQY6LSZ6Y3A, seg_pos={0 0}, upload=523642]: Done uploading piece. err: <nil> Dec 30 12:51:38.528133793 (597h22m38.399216838s) [seg_pos={0 0}, upload=523642]: Segment reached optimal threshold of 14 pieces. Dec 30 12:51:38.528364652 (597h22m38.399447687s) [noise=true , address=45.91.116.9:4289 , piece_id=O735F2PWWAXGOP54RXURLZ64S7XC4XZZY4OH7RM4N2UOIAA3LJTA, seg_pos={0 0}, upload=523642]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rfBAoVRdhYiRJFWqnamoRAFLxEUsk7tShJEG3jvhKDbcHwipD): protocol: expected piece hash; context canceled; EOF Dec 30 12:51:38.528373429 (597h22m38.399456464s) [noise=true , address=45.91.116.9:4289 , piece_id=O735F2PWWAXGOP54RXURLZ64S7XC4XZZY4OH7RM4N2UOIAA3LJTA, seg_pos={0 0}, upload=523642]: Long tail cancellation due to optimal threshold Dec 30 12:51:38.528672969 (597h22m38.399756014s) [noise=true , address=145.79.227.40:4072 , piece_id=ACK67AVKO5NK7IJKXGL2SLN4GGEKGW47WNTCZCPHIAKIVXYGFHVA, seg_pos={0 0}, upload=523642]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FSQUJNBPY52p2Wkg48DWPN3bdciabNacXcgDEHSwLRrnPG9ne): protocol: expected piece hash; context canceled; EOF Dec 30 12:51:38.528682567 (597h22m38.399765602s) [noise=true , address=145.79.227.40:4072 , piece_id=ACK67AVKO5NK7IJKXGL2SLN4GGEKGW47WNTCZCPHIAKIVXYGFHVA, seg_pos={0 0}, upload=523642]: Long tail cancellation due to optimal threshold Dec 30 12:51:38.528839085 (597h22m38.39992212s) [noise=true , address=77.90.31.11:4583 , piece_id=JEMHBN5CJQI4GTDPRB5RZLBRDUWB46JL7JTORRZXYJTLFK7Z66PA, seg_pos={0 0}, upload=523642]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Ttdx6LHmuAt8JVC7arCDgWjm9eLadjeaEw17F12btoXuSiq9t): protocol: expected piece hash; context canceled; EOF Dec 30 12:51:38.528847421 (597h22m38.399930476s) [noise=true , address=77.90.31.11:4583 , piece_id=JEMHBN5CJQI4GTDPRB5RZLBRDUWB46JL7JTORRZXYJTLFK7Z66PA, seg_pos={0 0}, upload=523642]: Long tail cancellation due to optimal threshold Dec 30 12:51:38.529001655 (597h22m38.4000847s) [noise=true , address=64.49.13.169:4716 , piece_id=EIRJU5OSYOF3D3AWCGKKSPFL36THXWUWJWIMBYHVEPI7VB6ATD3Q, seg_pos={0 0}, upload=523642]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1w21xeA7emhjphgYw2Qyz3uuHgPezuivmVD9KmrKmNZk3TnaGN): protocol: write tcp 188.166.225.170:59670->64.49.13.169:4716: use of closed network connection; write tcp 188.166.225.170:59670->64.49.13.169:4716: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59670->64.49.13.169:4716: use of closed network connection Dec 30 12:51:38.529010362 (597h22m38.400093407s) [noise=true , address=64.49.13.169:4716 , piece_id=EIRJU5OSYOF3D3AWCGKKSPFL36THXWUWJWIMBYHVEPI7VB6ATD3Q, seg_pos={0 0}, upload=523642]: Long tail cancellation due to optimal threshold Dec 30 12:51:38.529029568 (597h22m38.400112603s) [seg_pos={0 0}, upload=523642]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 12:51:38.529107577 (597h22m38.400190612s) [upload=523642]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 12:51:38.529114079 (597h22m38.400197124s) [upload=523642]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 12:51:38.828356312 (597h22m38.699439387s) [upload=523642]: Done uploading segments... Dec 30 12:51:38.828379166 (597h22m38.699462231s) [upload=523642]: Upload finished. err: <nil> Duration: 43.320764589s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 46 out of 60 is failed (threshold=6) [XX.XX.XXX...X.XXXXXXXXXXX.X.XXXXXXXXX.X.XXXXXX.X.XXX.XXXX.XX] duration: 5.000842117s trace: https://jaeger.dev.storj.io/trace/3b1d58855f5733b3 FAIL (2025-12-30 13:02:22 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.00086974s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [X..X.......X.XX..X..X.XX...............X..X.X..X....X.......] duration: 5.76026998s trace: https://jaeger.dev.storj.io/trace/23ad8207da0148f9 FAIL (2025-12-30 13:02:18 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, Piece ID: TIHHT6FYFWUDFIVQZVQLE5IAL5Q2E3TKHDDIUBZSGFBQKQS4W5GA) context deadline exceeded; piecestore: (Node ID: 1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, Piece ID: J27M26BILJ4V23GMBQGK4OVKICA4TE3LDARCMDFPH7W6RWBGUYKQ) context deadline exceeded ======LOGS====== ments... Dec 30 12:55:52.163563927 (597h26m52.034646982s) [upload=523705]: Got next segment. Inline: false Dec 30 12:55:52.163612819 (597h26m52.034695864s) [upload=523705]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 12:55:52.163618821 (597h26m52.034701866s) [upload=523705]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 12:55:52.410969244 (597h26m52.282052279s) [seg_pos={0 0}, upload=523705]: Begin upload segment... Dec 30 12:55:52.411037163 (597h26m52.282120198s) [seg_pos={0 0}, upload=523705]: Done begin upload segment. Dec 30 12:55:52.411066920 (597h26m52.282149955s) [noise=true , address=145.79.227.14:4838 , piece_id=AZX5QHDBV7AXA4FIKQ343QISBB3LH24ZHORG53CKKKGGXDXC4QWQ, seg_pos={0 0}, upload=523705]: Uploading piece... Dec 30 12:55:52.411202477 (597h26m52.282285522s) [noise=true , address=45.91.116.6:4973 , piece_id=OT2DELPCMUI22A444ULTMOO4CYN4VD32E654SR44MDN2WQUATGDA, seg_pos={0 0}, upload=523705]: Uploading piece... Dec 30 12:55:52.411721015 (597h26m52.28280405s) [noise=true , address=77.90.31.13:4626 , piece_id=TIHHT6FYFWUDFIVQZVQLE5IAL5Q2E3TKHDDIUBZSGFBQKQS4W5GA, seg_pos={0 0}, upload=523705]: Uploading piece... Dec 30 12:55:52.412140103 (597h26m52.283223138s) [noise=true , address=45.91.116.8:4048 , piece_id=UXYGO5E3UBXUJKN442OB6CJWQYGW6Y5XUGABC467V2EIZP4KILTA, seg_pos={0 0}, upload=523705]: Uploading piece... Dec 30 12:55:52.412521419 (597h26m52.283604454s) [noise=true , address=145.79.227.50:4373 , piece_id=BA2SH47KSKE7F5Q4N5HXVNYDZ5FR453LMZHUDBWZZZKYKOIRZXEQ, seg_pos={0 0}, upload=523705]: Uploading piece... Dec 30 12:55:52.412888228 (597h26m52.283971263s) [noise=true , address=145.79.227.27:4691 , piece_id=AG67HXKZBGSF3JV75G2RC5OXLUSKINQFPOOS2WNSR3R6P5FNPGLQ, seg_pos={0 0}, upload=523705]: Uploading piece... Dec 30 12:55:52.413237172 (597h26m52.284320207s) [noise=true , address=145.79.227.24:4581 , piece_id=XNJDC2HJ6NIEXHNMP33STICUMDD5PQ2Y7UGMWU65ZPNH27R5PVDQ, seg_pos={0 0}, upload=523705]: Uploading piece... Dec 30 12:55:52.413653144 (597h26m52.284736179s) [noise=true , address=145.79.227.21:4475 , piece_id=42PVG5Q7APUS6QHI5JP6H2OLYIJSN5FXFSTDCKSNZVRKYPWVNI6A, seg_pos={0 0}, upload=523705]: Uploading piece... Dec 30 12:55:52.414092440 (597h26m52.285175475s) [noise=true , address=45.91.116.13:4153 , piece_id=J27M26BILJ4V23GMBQGK4OVKICA4TE3LDARCMDFPH7W6RWBGUYKQ, seg_pos={0 0}, upload=523705]: Uploading piece... Dec 30 12:55:52.415787007 (597h26m52.286870042s) [noise=true , address=45.91.116.11:4118 , piece_id=2JG6PCUGTT3K6YUMTWNZQB3OXTDZ7RMAFYHHIJMHLVFDY2E5K75A, seg_pos={0 0}, upload=523705]: Uploading piece... Dec 30 12:55:52.415936882 (597h26m52.287019917s) [noise=true , address=45.91.116.14:4189 , piece_id=X7CQTC447CXK6ITDRX4FZN2DNAWCPAUDO2UYH7O2EDBBF2OS35JA, seg_pos={0 0}, upload=523705]: Uploading piece... Dec 30 12:55:52.416357675 (597h26m52.28744072s) [noise=true , address=77.90.31.10:4519 , piece_id=7Q52TQ7IO2B5RCQESR6HYMYW4CZRO5ZPDJPXEAFEKGDPW7FSWE4Q, seg_pos={0 0}, upload=523705]: Uploading piece... Dec 30 12:55:52.416794676 (597h26m52.287877721s) [noise=true , address=45.91.116.9:4262 , piece_id=NCLXSMOIPWB37ABGU6FUUSNFW5MUZ5PQT4YGT4YGCJ2QXNQ6ML6Q, seg_pos={0 0}, upload=523705]: Uploading piece... Dec 30 12:55:52.417238962 (597h26m52.288321997s) [noise=true , address=145.79.227.47:4267 , piece_id=YPMQM5AXQF3LHHNPVIQWNBUHILPO7ORH7P2DUOKG5LMWMXBNVGGA, seg_pos={0 0}, upload=523705]: Uploading piece... Dec 30 12:55:52.417715961 (597h26m52.288798996s) [noise=true , address=45.91.116.7:4012 , piece_id=NNXDGUVP755B6VQ7L6LFLRYE3KN7KDEMKHEUL7TS3GYPSBXHERHA, seg_pos={0 0}, upload=523705]: Uploading piece... Dec 30 12:55:52.418153584 (597h26m52.289236619s) [noise=true , address=145.79.227.20:4438 , piece_id=LRYZYP6O6RNIIPK2VLTX3A2IL34LBGFFOIINSAMZ3TY5LUNWTCKQ, seg_pos={0 0}, upload=523705]: Uploading piece... Dec 30 12:55:54.018730327 (597h26m53.889813362s) [upload=523705]: Next returned nil segment Dec 30 12:55:54.018743952 (597h26m53.889826997s) [upload=523705]: Waiting for error group managing segments... Dec 30 12:55:54.399128062 (597h26m54.270211097s) [noise=true , address=145.79.227.24:4581 , piece_id=XNJDC2HJ6NIEXHNMP33STICUMDD5PQ2Y7UGMWU65ZPNH27R5PVDQ, seg_pos={0 0}, upload=523705]: Done uploading piece. err: <nil> Dec 30 12:55:54.399875084 (597h26m54.270958129s) [noise=true , address=145.79.227.14:4838 , piece_id=AZX5QHDBV7AXA4FIKQ343QISBB3LH24ZHORG53CKKKGGXDXC4QWQ, seg_pos={0 0}, upload=523705]: Done uploading piece. err: <nil> Dec 30 12:55:54.404232521 (597h26m54.275315556s) [noise=true , address=145.79.227.27:4691 , piece_id=AG67HXKZBGSF3JV75G2RC5OXLUSKINQFPOOS2WNSR3R6P5FNPGLQ, seg_pos={0 0}, upload=523705]: Done uploading piece. err: <nil> Dec 30 12:55:54.424052321 (597h26m54.295135356s) [noise=true , address=145.79.227.20:4438 , piece_id=LRYZYP6O6RNIIPK2VLTX3A2IL34LBGFFOIINSAMZ3TY5LUNWTCKQ, seg_pos={0 0}, upload=523705]: Done uploading piece. err: <nil> Dec 30 12:55:54.433077286 (597h26m54.304160321s) [noise=true , address=145.79.227.21:4475 , piece_id=42PVG5Q7APUS6QHI5JP6H2OLYIJSN5FXFSTDCKSNZVRKYPWVNI6A, seg_pos={0 0}, upload=523705]: Done uploading piece. err: <nil> Dec 30 12:55:54.437149530 (597h26m54.308232575s) [noise=true , address=145.79.227.47:4267 , piece_id=YPMQM5AXQF3LHHNPVIQWNBUHILPO7ORH7P2DUOKG5LMWMXBNVGGA, seg_pos={0 0}, upload=523705]: Done uploading piece. err: <nil> Dec 30 12:55:54.480458237 (597h26m54.351541282s) [noise=true , address=145.79.227.50:4373 , piece_id=BA2SH47KSKE7F5Q4N5HXVNYDZ5FR453LMZHUDBWZZZKYKOIRZXEQ, seg_pos={0 0}, upload=523705]: Done uploading piece. err: <nil> Dec 30 12:55:54.689140229 (597h26m54.560223274s) [noise=true , address=77.90.31.13:4626 , piece_id=TIHHT6FYFWUDFIVQZVQLE5IAL5Q2E3TKHDDIUBZSGFBQKQS4W5GA, seg_pos={0 0}, upload=523705]: Done uploading piece. err: <nil> Dec 30 12:55:54.711163926 (597h26m54.582246961s) [noise=true , address=45.91.116.13:4153 , piece_id=J27M26BILJ4V23GMBQGK4OVKICA4TE3LDARCMDFPH7W6RWBGUYKQ, seg_pos={0 0}, upload=523705]: Done uploading piece. err: <nil> Dec 30 12:55:54.730262152 (597h26m54.601345197s) [noise=true , address=45.91.116.11:4118 , piece_id=2JG6PCUGTT3K6YUMTWNZQB3OXTDZ7RMAFYHHIJMHLVFDY2E5K75A, seg_pos={0 0}, upload=523705]: Done uploading piece. err: <nil> Dec 30 12:55:54.877886751 (597h26m54.748969796s) [noise=true , address=77.90.31.10:4519 , piece_id=7Q52TQ7IO2B5RCQESR6HYMYW4CZRO5ZPDJPXEAFEKGDPW7FSWE4Q, seg_pos={0 0}, upload=523705]: Done uploading piece. err: <nil> Dec 30 12:55:54.889062301 (597h26m54.760145336s) [noise=true , address=45.91.116.8:4048 , piece_id=UXYGO5E3UBXUJKN442OB6CJWQYGW6Y5XUGABC467V2EIZP4KILTA, seg_pos={0 0}, upload=523705]: Done uploading piece. err: <nil> Dec 30 12:55:55.074468032 (597h26m54.945551077s) [noise=true , address=45.91.116.9:4262 , piece_id=NCLXSMOIPWB37ABGU6FUUSNFW5MUZ5PQT4YGT4YGCJ2QXNQ6ML6Q, seg_pos={0 0}, upload=523705]: Done uploading piece. err: <nil> Dec 30 12:55:55.074489473 (597h26m54.945572518s) [seg_pos={0 0}, upload=523705]: Segment reached optimal threshold of 13 pieces. Dec 30 12:55:55.074727766 (597h26m54.945810811s) [noise=true , address=45.91.116.14:4189 , piece_id=X7CQTC447CXK6ITDRX4FZN2DNAWCPAUDO2UYH7O2EDBBF2OS35JA, seg_pos={0 0}, upload=523705]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: write tcp 188.166.225.170:57284->45.91.116.14:4189: use of closed network connection; write tcp 188.166.225.170:57284->45.91.116.14:4189: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57284->45.91.116.14:4189: use of closed network connection Dec 30 12:55:55.074735361 (597h26m54.945818396s) [noise=true , address=45.91.116.14:4189 , piece_id=X7CQTC447CXK6ITDRX4FZN2DNAWCPAUDO2UYH7O2EDBBF2OS35JA, seg_pos={0 0}, upload=523705]: Long tail cancellation due to optimal threshold Dec 30 12:55:55.074903922 (597h26m54.945986967s) [noise=true , address=45.91.116.7:4012 , piece_id=NNXDGUVP755B6VQ7L6LFLRYE3KN7KDEMKHEUL7TS3GYPSBXHERHA, seg_pos={0 0}, upload=523705]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t): protocol: write tcp 188.166.225.170:49534->45.91.116.7:4012: use of closed network connection; write tcp 188.166.225.170:49534->45.91.116.7:4012: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49534->45.91.116.7:4012: use of closed network connection Dec 30 12:55:55.074929440 (597h26m54.946012485s) [noise=true , address=45.91.116.7:4012 , piece_id=NNXDGUVP755B6VQ7L6LFLRYE3KN7KDEMKHEUL7TS3GYPSBXHERHA, seg_pos={0 0}, upload=523705]: Long tail cancellation due to optimal threshold Dec 30 12:55:55.074969206 (597h26m54.946052241s) [noise=true , address=45.91.116.6:4973 , piece_id=OT2DELPCMUI22A444ULTMOO4CYN4VD32E654SR44MDN2WQUATGDA, seg_pos={0 0}, upload=523705]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV): protocol: expected piece hash; context canceled; EOF Dec 30 12:55:55.074975178 (597h26m54.946058213s) [noise=true , address=45.91.116.6:4973 , piece_id=OT2DELPCMUI22A444ULTMOO4CYN4VD32E654SR44MDN2WQUATGDA, seg_pos={0 0}, upload=523705]: Long tail cancellation due to optimal threshold Dec 30 12:55:55.075042837 (597h26m54.946125872s) [seg_pos={0 0}, upload=523705]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 12:55:55.075106016 (597h26m54.946189051s) [upload=523705]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 12:55:55.075110695 (597h26m54.94619373s) [upload=523705]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 12:55:55.365579192 (597h26m55.236662227s) [upload=523705]: Done uploading segments... Dec 30 12:55:55.365601594 (597h26m55.236684639s) [upload=523705]: Upload finished. err: <nil> Duration: 5.76029667s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [..............................X........X...........X..X.....] duration: 11.768955234s trace: https://jaeger.dev.storj.io/trace/17a3dfac1fc2c808 FAIL (2025-12-30 13:02:17 UTC) orbiter warning: context deadline exceeded Duration: 11.768970113s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [.................X.......................X....X.............] duration: 12.01991568s trace: https://jaeger.dev.storj.io/trace/7b47a373389cd7c3 FAIL (2025-12-30 13:01:51 UTC) orbiter warning: context deadline exceeded Duration: 12.019945136s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [XX..X.......X.XX..X..X.XX...............X..X.X..X....X......] duration: 5.657716735s trace: https://jaeger.dev.storj.io/trace/23ad8207da0148f9 FAIL (2025-12-30 13:01:47 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, Piece ID: TIHHT6FYFWUDFIVQZVQLE5IAL5Q2E3TKHDDIUBZSGFBQKQS4W5GA) context deadline exceeded; piecestore: (Node ID: 1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, Piece ID: J27M26BILJ4V23GMBQGK4OVKICA4TE3LDARCMDFPH7W6RWBGUYKQ) context deadline exceeded ======LOGS====== ments... Dec 30 12:55:52.163563927 (597h26m52.034646982s) [upload=523705]: Got next segment. Inline: false Dec 30 12:55:52.163612819 (597h26m52.034695864s) [upload=523705]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 12:55:52.163618821 (597h26m52.034701866s) [upload=523705]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 12:55:52.410969244 (597h26m52.282052279s) [seg_pos={0 0}, upload=523705]: Begin upload segment... Dec 30 12:55:52.411037163 (597h26m52.282120198s) [seg_pos={0 0}, upload=523705]: Done begin upload segment. Dec 30 12:55:52.411066920 (597h26m52.282149955s) [noise=true , address=145.79.227.14:4838 , piece_id=AZX5QHDBV7AXA4FIKQ343QISBB3LH24ZHORG53CKKKGGXDXC4QWQ, seg_pos={0 0}, upload=523705]: Uploading piece... Dec 30 12:55:52.411202477 (597h26m52.282285522s) [noise=true , address=45.91.116.6:4973 , piece_id=OT2DELPCMUI22A444ULTMOO4CYN4VD32E654SR44MDN2WQUATGDA, seg_pos={0 0}, upload=523705]: Uploading piece... Dec 30 12:55:52.411721015 (597h26m52.28280405s) [noise=true , address=77.90.31.13:4626 , piece_id=TIHHT6FYFWUDFIVQZVQLE5IAL5Q2E3TKHDDIUBZSGFBQKQS4W5GA, seg_pos={0 0}, upload=523705]: Uploading piece... Dec 30 12:55:52.412140103 (597h26m52.283223138s) [noise=true , address=45.91.116.8:4048 , piece_id=UXYGO5E3UBXUJKN442OB6CJWQYGW6Y5XUGABC467V2EIZP4KILTA, seg_pos={0 0}, upload=523705]: Uploading piece... Dec 30 12:55:52.412521419 (597h26m52.283604454s) [noise=true , address=145.79.227.50:4373 , piece_id=BA2SH47KSKE7F5Q4N5HXVNYDZ5FR453LMZHUDBWZZZKYKOIRZXEQ, seg_pos={0 0}, upload=523705]: Uploading piece... Dec 30 12:55:52.412888228 (597h26m52.283971263s) [noise=true , address=145.79.227.27:4691 , piece_id=AG67HXKZBGSF3JV75G2RC5OXLUSKINQFPOOS2WNSR3R6P5FNPGLQ, seg_pos={0 0}, upload=523705]: Uploading piece... Dec 30 12:55:52.413237172 (597h26m52.284320207s) [noise=true , address=145.79.227.24:4581 , piece_id=XNJDC2HJ6NIEXHNMP33STICUMDD5PQ2Y7UGMWU65ZPNH27R5PVDQ, seg_pos={0 0}, upload=523705]: Uploading piece... Dec 30 12:55:52.413653144 (597h26m52.284736179s) [noise=true , address=145.79.227.21:4475 , piece_id=42PVG5Q7APUS6QHI5JP6H2OLYIJSN5FXFSTDCKSNZVRKYPWVNI6A, seg_pos={0 0}, upload=523705]: Uploading piece... Dec 30 12:55:52.414092440 (597h26m52.285175475s) [noise=true , address=45.91.116.13:4153 , piece_id=J27M26BILJ4V23GMBQGK4OVKICA4TE3LDARCMDFPH7W6RWBGUYKQ, seg_pos={0 0}, upload=523705]: Uploading piece... Dec 30 12:55:52.415787007 (597h26m52.286870042s) [noise=true , address=45.91.116.11:4118 , piece_id=2JG6PCUGTT3K6YUMTWNZQB3OXTDZ7RMAFYHHIJMHLVFDY2E5K75A, seg_pos={0 0}, upload=523705]: Uploading piece... Dec 30 12:55:52.415936882 (597h26m52.287019917s) [noise=true , address=45.91.116.14:4189 , piece_id=X7CQTC447CXK6ITDRX4FZN2DNAWCPAUDO2UYH7O2EDBBF2OS35JA, seg_pos={0 0}, upload=523705]: Uploading piece... Dec 30 12:55:52.416357675 (597h26m52.28744072s) [noise=true , address=77.90.31.10:4519 , piece_id=7Q52TQ7IO2B5RCQESR6HYMYW4CZRO5ZPDJPXEAFEKGDPW7FSWE4Q, seg_pos={0 0}, upload=523705]: Uploading piece... Dec 30 12:55:52.416794676 (597h26m52.287877721s) [noise=true , address=45.91.116.9:4262 , piece_id=NCLXSMOIPWB37ABGU6FUUSNFW5MUZ5PQT4YGT4YGCJ2QXNQ6ML6Q, seg_pos={0 0}, upload=523705]: Uploading piece... Dec 30 12:55:52.417238962 (597h26m52.288321997s) [noise=true , address=145.79.227.47:4267 , piece_id=YPMQM5AXQF3LHHNPVIQWNBUHILPO7ORH7P2DUOKG5LMWMXBNVGGA, seg_pos={0 0}, upload=523705]: Uploading piece... Dec 30 12:55:52.417715961 (597h26m52.288798996s) [noise=true , address=45.91.116.7:4012 , piece_id=NNXDGUVP755B6VQ7L6LFLRYE3KN7KDEMKHEUL7TS3GYPSBXHERHA, seg_pos={0 0}, upload=523705]: Uploading piece... Dec 30 12:55:52.418153584 (597h26m52.289236619s) [noise=true , address=145.79.227.20:4438 , piece_id=LRYZYP6O6RNIIPK2VLTX3A2IL34LBGFFOIINSAMZ3TY5LUNWTCKQ, seg_pos={0 0}, upload=523705]: Uploading piece... Dec 30 12:55:54.018730327 (597h26m53.889813362s) [upload=523705]: Next returned nil segment Dec 30 12:55:54.018743952 (597h26m53.889826997s) [upload=523705]: Waiting for error group managing segments... Dec 30 12:55:54.399128062 (597h26m54.270211097s) [noise=true , address=145.79.227.24:4581 , piece_id=XNJDC2HJ6NIEXHNMP33STICUMDD5PQ2Y7UGMWU65ZPNH27R5PVDQ, seg_pos={0 0}, upload=523705]: Done uploading piece. err: <nil> Dec 30 12:55:54.399875084 (597h26m54.270958129s) [noise=true , address=145.79.227.14:4838 , piece_id=AZX5QHDBV7AXA4FIKQ343QISBB3LH24ZHORG53CKKKGGXDXC4QWQ, seg_pos={0 0}, upload=523705]: Done uploading piece. err: <nil> Dec 30 12:55:54.404232521 (597h26m54.275315556s) [noise=true , address=145.79.227.27:4691 , piece_id=AG67HXKZBGSF3JV75G2RC5OXLUSKINQFPOOS2WNSR3R6P5FNPGLQ, seg_pos={0 0}, upload=523705]: Done uploading piece. err: <nil> Dec 30 12:55:54.424052321 (597h26m54.295135356s) [noise=true , address=145.79.227.20:4438 , piece_id=LRYZYP6O6RNIIPK2VLTX3A2IL34LBGFFOIINSAMZ3TY5LUNWTCKQ, seg_pos={0 0}, upload=523705]: Done uploading piece. err: <nil> Dec 30 12:55:54.433077286 (597h26m54.304160321s) [noise=true , address=145.79.227.21:4475 , piece_id=42PVG5Q7APUS6QHI5JP6H2OLYIJSN5FXFSTDCKSNZVRKYPWVNI6A, seg_pos={0 0}, upload=523705]: Done uploading piece. err: <nil> Dec 30 12:55:54.437149530 (597h26m54.308232575s) [noise=true , address=145.79.227.47:4267 , piece_id=YPMQM5AXQF3LHHNPVIQWNBUHILPO7ORH7P2DUOKG5LMWMXBNVGGA, seg_pos={0 0}, upload=523705]: Done uploading piece. err: <nil> Dec 30 12:55:54.480458237 (597h26m54.351541282s) [noise=true , address=145.79.227.50:4373 , piece_id=BA2SH47KSKE7F5Q4N5HXVNYDZ5FR453LMZHUDBWZZZKYKOIRZXEQ, seg_pos={0 0}, upload=523705]: Done uploading piece. err: <nil> Dec 30 12:55:54.689140229 (597h26m54.560223274s) [noise=true , address=77.90.31.13:4626 , piece_id=TIHHT6FYFWUDFIVQZVQLE5IAL5Q2E3TKHDDIUBZSGFBQKQS4W5GA, seg_pos={0 0}, upload=523705]: Done uploading piece. err: <nil> Dec 30 12:55:54.711163926 (597h26m54.582246961s) [noise=true , address=45.91.116.13:4153 , piece_id=J27M26BILJ4V23GMBQGK4OVKICA4TE3LDARCMDFPH7W6RWBGUYKQ, seg_pos={0 0}, upload=523705]: Done uploading piece. err: <nil> Dec 30 12:55:54.730262152 (597h26m54.601345197s) [noise=true , address=45.91.116.11:4118 , piece_id=2JG6PCUGTT3K6YUMTWNZQB3OXTDZ7RMAFYHHIJMHLVFDY2E5K75A, seg_pos={0 0}, upload=523705]: Done uploading piece. err: <nil> Dec 30 12:55:54.877886751 (597h26m54.748969796s) [noise=true , address=77.90.31.10:4519 , piece_id=7Q52TQ7IO2B5RCQESR6HYMYW4CZRO5ZPDJPXEAFEKGDPW7FSWE4Q, seg_pos={0 0}, upload=523705]: Done uploading piece. err: <nil> Dec 30 12:55:54.889062301 (597h26m54.760145336s) [noise=true , address=45.91.116.8:4048 , piece_id=UXYGO5E3UBXUJKN442OB6CJWQYGW6Y5XUGABC467V2EIZP4KILTA, seg_pos={0 0}, upload=523705]: Done uploading piece. err: <nil> Dec 30 12:55:55.074468032 (597h26m54.945551077s) [noise=true , address=45.91.116.9:4262 , piece_id=NCLXSMOIPWB37ABGU6FUUSNFW5MUZ5PQT4YGT4YGCJ2QXNQ6ML6Q, seg_pos={0 0}, upload=523705]: Done uploading piece. err: <nil> Dec 30 12:55:55.074489473 (597h26m54.945572518s) [seg_pos={0 0}, upload=523705]: Segment reached optimal threshold of 13 pieces. Dec 30 12:55:55.074727766 (597h26m54.945810811s) [noise=true , address=45.91.116.14:4189 , piece_id=X7CQTC447CXK6ITDRX4FZN2DNAWCPAUDO2UYH7O2EDBBF2OS35JA, seg_pos={0 0}, upload=523705]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: write tcp 188.166.225.170:57284->45.91.116.14:4189: use of closed network connection; write tcp 188.166.225.170:57284->45.91.116.14:4189: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57284->45.91.116.14:4189: use of closed network connection Dec 30 12:55:55.074735361 (597h26m54.945818396s) [noise=true , address=45.91.116.14:4189 , piece_id=X7CQTC447CXK6ITDRX4FZN2DNAWCPAUDO2UYH7O2EDBBF2OS35JA, seg_pos={0 0}, upload=523705]: Long tail cancellation due to optimal threshold Dec 30 12:55:55.074903922 (597h26m54.945986967s) [noise=true , address=45.91.116.7:4012 , piece_id=NNXDGUVP755B6VQ7L6LFLRYE3KN7KDEMKHEUL7TS3GYPSBXHERHA, seg_pos={0 0}, upload=523705]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t): protocol: write tcp 188.166.225.170:49534->45.91.116.7:4012: use of closed network connection; write tcp 188.166.225.170:49534->45.91.116.7:4012: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49534->45.91.116.7:4012: use of closed network connection Dec 30 12:55:55.074929440 (597h26m54.946012485s) [noise=true , address=45.91.116.7:4012 , piece_id=NNXDGUVP755B6VQ7L6LFLRYE3KN7KDEMKHEUL7TS3GYPSBXHERHA, seg_pos={0 0}, upload=523705]: Long tail cancellation due to optimal threshold Dec 30 12:55:55.074969206 (597h26m54.946052241s) [noise=true , address=45.91.116.6:4973 , piece_id=OT2DELPCMUI22A444ULTMOO4CYN4VD32E654SR44MDN2WQUATGDA, seg_pos={0 0}, upload=523705]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV): protocol: expected piece hash; context canceled; EOF Dec 30 12:55:55.074975178 (597h26m54.946058213s) [noise=true , address=45.91.116.6:4973 , piece_id=OT2DELPCMUI22A444ULTMOO4CYN4VD32E654SR44MDN2WQUATGDA, seg_pos={0 0}, upload=523705]: Long tail cancellation due to optimal threshold Dec 30 12:55:55.075042837 (597h26m54.946125872s) [seg_pos={0 0}, upload=523705]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 12:55:55.075106016 (597h26m54.946189051s) [upload=523705]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 12:55:55.075110695 (597h26m54.94619373s) [upload=523705]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 12:55:55.365579192 (597h26m55.236662227s) [upload=523705]: Done uploading segments... Dec 30 12:55:55.365601594 (597h26m55.236684639s) [upload=523705]: Upload finished. err: <nil> Duration: 5.65775631s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 7 out of 60 is failed (threshold=3) [............X.....X.X...X.........X....X........X...........] duration: 29.45699545s trace: https://jaeger.dev.storj.io/trace/1e6433ec4a141391 FAIL (2025-12-30 13:01:36 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== [noise=true , address=77.90.31.11:4583 , piece_id=JEMHBN5CJQI4GTDPRB5RZLBRDUWB46JL7JTORRZXYJTLFK7Z66PA, seg_pos={0 0}, upload=523642]: Uploading piece... Dec 30 12:51:35.352305080 (597h22m35.223388115s) [noise=true , address=145.79.227.44:4222 , piece_id=32QNUAYLXL4AQYBRI6LDICOWQIZMSU3OVN5SWVOCB6XIZY4FHRSA, seg_pos={0 0}, upload=523642]: Uploading piece... Dec 30 12:51:35.353083031 (597h22m35.224166066s) [noise=true , address=145.79.227.34:4925 , piece_id=G46JE4KCDC7FPDEUW2YDTSLI5X66MAMDCRRLDG7EBYF6ENLELLEQ, seg_pos={0 0}, upload=523642]: Uploading piece... Dec 30 12:51:35.353482140 (597h22m35.224565175s) [noise=true , address=145.79.227.33:4897 , piece_id=6SGUOB6IMRF2SPO6WHSKB6XLQENVS5TCJ6RMO6BYIMQ2PANMHK6A, seg_pos={0 0}, upload=523642]: Uploading piece... Dec 30 12:51:35.354315267 (597h22m35.225398302s) [noise=true , address=145.79.227.17:4991 , piece_id=GUPSOAIXZGPRJ7MVZ46464IRG7FBQEFR2RHYDT2GIPAMAKN36YHQ, seg_pos={0 0}, upload=523642]: Uploading piece... Dec 30 12:51:35.354723675 (597h22m35.22580671s) [noise=true , address=145.79.227.2:4119 , piece_id=55AC2GONNJPSGZJ3Y3CYFRLUVKRDMZGLY4HMAPDMEGMN6AJCYWOQ, seg_pos={0 0}, upload=523642]: Uploading piece... Dec 30 12:51:35.355352552 (597h22m35.226435587s) [noise=true , address=64.49.13.169:4716 , piece_id=EIRJU5OSYOF3D3AWCGKKSPFL36THXWUWJWIMBYHVEPI7VB6ATD3Q, seg_pos={0 0}, upload=523642]: Uploading piece... Dec 30 12:51:35.356361994 (597h22m35.227445029s) [noise=true , address=145.79.227.49:4367 , piece_id=6L53GGMES5PBPHPLUCPDUGYKKOQQ64VSZWAGX6EP755TJXT5RH6A, seg_pos={0 0}, upload=523642]: Uploading piece... Dec 30 12:51:35.356578216 (597h22m35.227661261s) [noise=true , address=77.90.31.14:4682 , piece_id=XACXXUOJEEGBXQEJFZYOZKQDX5N2V3LX5NCEQP64YIBZPHK23NQA, seg_pos={0 0}, upload=523642]: Uploading piece... Dec 30 12:51:35.357840370 (597h22m35.228923405s) [noise=true , address=145.79.227.46:4612 , piece_id=VG2ZGVJNHBN2GT72UXK6QA32HYGW6465MEOII6GYSOS7SZYLBBJA, seg_pos={0 0}, upload=523642]: Uploading piece... Dec 30 12:51:35.359039433 (597h22m35.230122468s) [noise=true , address=145.79.227.40:4072 , piece_id=ACK67AVKO5NK7IJKXGL2SLN4GGEKGW47WNTCZCPHIAKIVXYGFHVA, seg_pos={0 0}, upload=523642]: Uploading piece... Dec 30 12:51:35.359333493 (597h22m35.230416538s) [noise=true , address=77.90.31.18:4812 , piece_id=V347IHYDQTVIQ43JTHQJJW2BU3E4VSOSWABCPJPVX2JVCR6J6UNA, seg_pos={0 0}, upload=523642]: Uploading piece... Dec 30 12:51:35.360694154 (597h22m35.231777189s) [noise=true , address=45.91.116.9:4289 , piece_id=O735F2PWWAXGOP54RXURLZ64S7XC4XZZY4OH7RM4N2UOIAA3LJTA, seg_pos={0 0}, upload=523642]: Uploading piece... Dec 30 12:51:35.361764952 (597h22m35.232847987s) [noise=true , address=145.79.227.43:4160 , piece_id=746FBAMI37I2B2UMEKDGOVGNDHN5EXJJH6CMGSCTEEOAVLNDXVXA, seg_pos={0 0}, upload=523642]: Uploading piece... Dec 30 12:51:35.363365710 (597h22m35.234448755s) [noise=true , address=145.79.227.19:4099 , piece_id=DLLN7EUVR2D72XD7BL57AW5JPHVE76PUWF6LWXDEZQH6JZVH3RZQ, seg_pos={0 0}, upload=523642]: Uploading piece... Dec 30 12:51:35.363684648 (597h22m35.234767683s) [noise=true , address=145.79.227.30:4775 , piece_id=JADWMGGTCCSYQMINZV4CLHKEAWREFOS5S4RSY2MFBKC3ULFBD7FA, seg_pos={0 0}, upload=523642]: Uploading piece... Dec 30 12:51:36.749009517 (597h22m36.620092552s) [upload=523642]: Next returned nil segment Dec 30 12:51:36.749023393 (597h22m36.620106428s) [upload=523642]: Waiting for error group managing segments... Dec 30 12:51:37.208596330 (597h22m37.079679376s) [noise=true , address=145.79.227.17:4991 , piece_id=GUPSOAIXZGPRJ7MVZ46464IRG7FBQEFR2RHYDT2GIPAMAKN36YHQ, seg_pos={0 0}, upload=523642]: Done uploading piece. err: <nil> Dec 30 12:51:37.240881687 (597h22m37.111964722s) [noise=true , address=145.79.227.49:4367 , piece_id=6L53GGMES5PBPHPLUCPDUGYKKOQQ64VSZWAGX6EP755TJXT5RH6A, seg_pos={0 0}, upload=523642]: Done uploading piece. err: <nil> Dec 30 12:51:37.294293140 (597h22m37.165376175s) [noise=true , address=145.79.227.2:4119 , piece_id=55AC2GONNJPSGZJ3Y3CYFRLUVKRDMZGLY4HMAPDMEGMN6AJCYWOQ, seg_pos={0 0}, upload=523642]: Done uploading piece. err: <nil> Dec 30 12:51:37.302040142 (597h22m37.173123177s) [noise=true , address=145.79.227.43:4160 , piece_id=746FBAMI37I2B2UMEKDGOVGNDHN5EXJJH6CMGSCTEEOAVLNDXVXA, seg_pos={0 0}, upload=523642]: Done uploading piece. err: <nil> Dec 30 12:51:37.308919160 (597h22m37.180002196s) [noise=true , address=145.79.227.19:4099 , piece_id=DLLN7EUVR2D72XD7BL57AW5JPHVE76PUWF6LWXDEZQH6JZVH3RZQ, seg_pos={0 0}, upload=523642]: Done uploading piece. err: <nil> Dec 30 12:51:37.370286944 (597h22m37.24136999s) [noise=true , address=77.90.31.10:4545 , piece_id=YQV3IPXMIBSO5ZCISFUBGRGHE6FIXLKTLNYKPWR42VNBKNOGPG5Q, seg_pos={0 0}, upload=523642]: Done uploading piece. err: <nil> Dec 30 12:51:37.397564325 (597h22m37.26864737s) [noise=true , address=77.90.31.18:4812 , piece_id=V347IHYDQTVIQ43JTHQJJW2BU3E4VSOSWABCPJPVX2JVCR6J6UNA, seg_pos={0 0}, upload=523642]: Done uploading piece. err: <nil> Dec 30 12:51:37.481372759 (597h22m37.352455794s) [noise=true , address=77.90.31.14:4682 , piece_id=XACXXUOJEEGBXQEJFZYOZKQDX5N2V3LX5NCEQP64YIBZPHK23NQA, seg_pos={0 0}, upload=523642]: Done uploading piece. err: <nil> Dec 30 12:51:37.868609147 (597h22m37.739692192s) [noise=true , address=145.79.227.46:4612 , piece_id=VG2ZGVJNHBN2GT72UXK6QA32HYGW6465MEOII6GYSOS7SZYLBBJA, seg_pos={0 0}, upload=523642]: Done uploading piece. err: <nil> Dec 30 12:51:38.122711215 (597h22m37.993794251s) [noise=true , address=145.79.227.44:4222 , piece_id=32QNUAYLXL4AQYBRI6LDICOWQIZMSU3OVN5SWVOCB6XIZY4FHRSA, seg_pos={0 0}, upload=523642]: Done uploading piece. err: <nil> Dec 30 12:51:38.156332275 (597h22m38.02741531s) [noise=true , address=145.79.227.30:4775 , piece_id=JADWMGGTCCSYQMINZV4CLHKEAWREFOS5S4RSY2MFBKC3ULFBD7FA, seg_pos={0 0}, upload=523642]: Done uploading piece. err: <nil> Dec 30 12:51:38.210830938 (597h22m38.081913983s) [noise=true , address=145.79.227.34:4925 , piece_id=G46JE4KCDC7FPDEUW2YDTSLI5X66MAMDCRRLDG7EBYF6ENLELLEQ, seg_pos={0 0}, upload=523642]: Done uploading piece. err: <nil> Dec 30 12:51:38.318916249 (597h22m38.189999284s) [noise=true , address=145.79.227.33:4897 , piece_id=6SGUOB6IMRF2SPO6WHSKB6XLQENVS5TCJ6RMO6BYIMQ2PANMHK6A, seg_pos={0 0}, upload=523642]: Done uploading piece. err: <nil> Dec 30 12:51:38.528107081 (597h22m38.399190127s) [noise=true , address=193.149.164.143:4694 , piece_id=AOZVLCCSBEYZX3WES7JJVOA2BECEBPII3A33PG5KCSQY6LSZ6Y3A, seg_pos={0 0}, upload=523642]: Done uploading piece. err: <nil> Dec 30 12:51:38.528133793 (597h22m38.399216838s) [seg_pos={0 0}, upload=523642]: Segment reached optimal threshold of 14 pieces. Dec 30 12:51:38.528364652 (597h22m38.399447687s) [noise=true , address=45.91.116.9:4289 , piece_id=O735F2PWWAXGOP54RXURLZ64S7XC4XZZY4OH7RM4N2UOIAA3LJTA, seg_pos={0 0}, upload=523642]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rfBAoVRdhYiRJFWqnamoRAFLxEUsk7tShJEG3jvhKDbcHwipD): protocol: expected piece hash; context canceled; EOF Dec 30 12:51:38.528373429 (597h22m38.399456464s) [noise=true , address=45.91.116.9:4289 , piece_id=O735F2PWWAXGOP54RXURLZ64S7XC4XZZY4OH7RM4N2UOIAA3LJTA, seg_pos={0 0}, upload=523642]: Long tail cancellation due to optimal threshold Dec 30 12:51:38.528672969 (597h22m38.399756014s) [noise=true , address=145.79.227.40:4072 , piece_id=ACK67AVKO5NK7IJKXGL2SLN4GGEKGW47WNTCZCPHIAKIVXYGFHVA, seg_pos={0 0}, upload=523642]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FSQUJNBPY52p2Wkg48DWPN3bdciabNacXcgDEHSwLRrnPG9ne): protocol: expected piece hash; context canceled; EOF Dec 30 12:51:38.528682567 (597h22m38.399765602s) [noise=true , address=145.79.227.40:4072 , piece_id=ACK67AVKO5NK7IJKXGL2SLN4GGEKGW47WNTCZCPHIAKIVXYGFHVA, seg_pos={0 0}, upload=523642]: Long tail cancellation due to optimal threshold Dec 30 12:51:38.528839085 (597h22m38.39992212s) [noise=true , address=77.90.31.11:4583 , piece_id=JEMHBN5CJQI4GTDPRB5RZLBRDUWB46JL7JTORRZXYJTLFK7Z66PA, seg_pos={0 0}, upload=523642]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Ttdx6LHmuAt8JVC7arCDgWjm9eLadjeaEw17F12btoXuSiq9t): protocol: expected piece hash; context canceled; EOF Dec 30 12:51:38.528847421 (597h22m38.399930476s) [noise=true , address=77.90.31.11:4583 , piece_id=JEMHBN5CJQI4GTDPRB5RZLBRDUWB46JL7JTORRZXYJTLFK7Z66PA, seg_pos={0 0}, upload=523642]: Long tail cancellation due to optimal threshold Dec 30 12:51:38.529001655 (597h22m38.4000847s) [noise=true , address=64.49.13.169:4716 , piece_id=EIRJU5OSYOF3D3AWCGKKSPFL36THXWUWJWIMBYHVEPI7VB6ATD3Q, seg_pos={0 0}, upload=523642]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1w21xeA7emhjphgYw2Qyz3uuHgPezuivmVD9KmrKmNZk3TnaGN): protocol: write tcp 188.166.225.170:59670->64.49.13.169:4716: use of closed network connection; write tcp 188.166.225.170:59670->64.49.13.169:4716: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59670->64.49.13.169:4716: use of closed network connection Dec 30 12:51:38.529010362 (597h22m38.400093407s) [noise=true , address=64.49.13.169:4716 , piece_id=EIRJU5OSYOF3D3AWCGKKSPFL36THXWUWJWIMBYHVEPI7VB6ATD3Q, seg_pos={0 0}, upload=523642]: Long tail cancellation due to optimal threshold Dec 30 12:51:38.529029568 (597h22m38.400112603s) [seg_pos={0 0}, upload=523642]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 12:51:38.529107577 (597h22m38.400190612s) [upload=523642]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 12:51:38.529114079 (597h22m38.400197124s) [upload=523642]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 12:51:38.828356312 (597h22m38.699439387s) [upload=523642]: Done uploading segments... Dec 30 12:51:38.828379166 (597h22m38.699462231s) [upload=523642]: Upload finished. err: <nil> Duration: 29.457021408s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [...............................X........X...........X..X....] duration: 15.700676517s trace: https://jaeger.dev.storj.io/trace/17a3dfac1fc2c808 FAIL (2025-12-30 13:01:30 UTC) orbiter warning: context deadline exceeded Duration: 15.700701794s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 46 out of 60 is failed (threshold=6) [XXX.XX.XXX...X.XXXXXXXXXXX.X.XXXXXXXXX.X.XXXXXX.X.XXX.XXXX.X] duration: 5.001084467s trace: https://jaeger.dev.storj.io/trace/4a194d3d63581f75 FAIL (2025-12-30 13:01:19 UTC) orbiter warning: context deadline exceeded Duration: 5.001110125s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [.XX..X.......X.XX..X..X.XX...............X..X.X..X....X.....] duration: 28.381322226s trace: https://jaeger.dev.storj.io/trace/23ad8207da0148f9 FAIL (2025-12-30 13:01:18 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, Piece ID: TIHHT6FYFWUDFIVQZVQLE5IAL5Q2E3TKHDDIUBZSGFBQKQS4W5GA) context deadline exceeded; piecestore: (Node ID: 1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, Piece ID: J27M26BILJ4V23GMBQGK4OVKICA4TE3LDARCMDFPH7W6RWBGUYKQ) context deadline exceeded ======LOGS====== ments... Dec 30 12:55:52.163563927 (597h26m52.034646982s) [upload=523705]: Got next segment. Inline: false Dec 30 12:55:52.163612819 (597h26m52.034695864s) [upload=523705]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 12:55:52.163618821 (597h26m52.034701866s) [upload=523705]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 12:55:52.410969244 (597h26m52.282052279s) [seg_pos={0 0}, upload=523705]: Begin upload segment... Dec 30 12:55:52.411037163 (597h26m52.282120198s) [seg_pos={0 0}, upload=523705]: Done begin upload segment. Dec 30 12:55:52.411066920 (597h26m52.282149955s) [noise=true , address=145.79.227.14:4838 , piece_id=AZX5QHDBV7AXA4FIKQ343QISBB3LH24ZHORG53CKKKGGXDXC4QWQ, seg_pos={0 0}, upload=523705]: Uploading piece... Dec 30 12:55:52.411202477 (597h26m52.282285522s) [noise=true , address=45.91.116.6:4973 , piece_id=OT2DELPCMUI22A444ULTMOO4CYN4VD32E654SR44MDN2WQUATGDA, seg_pos={0 0}, upload=523705]: Uploading piece... Dec 30 12:55:52.411721015 (597h26m52.28280405s) [noise=true , address=77.90.31.13:4626 , piece_id=TIHHT6FYFWUDFIVQZVQLE5IAL5Q2E3TKHDDIUBZSGFBQKQS4W5GA, seg_pos={0 0}, upload=523705]: Uploading piece... Dec 30 12:55:52.412140103 (597h26m52.283223138s) [noise=true , address=45.91.116.8:4048 , piece_id=UXYGO5E3UBXUJKN442OB6CJWQYGW6Y5XUGABC467V2EIZP4KILTA, seg_pos={0 0}, upload=523705]: Uploading piece... Dec 30 12:55:52.412521419 (597h26m52.283604454s) [noise=true , address=145.79.227.50:4373 , piece_id=BA2SH47KSKE7F5Q4N5HXVNYDZ5FR453LMZHUDBWZZZKYKOIRZXEQ, seg_pos={0 0}, upload=523705]: Uploading piece... Dec 30 12:55:52.412888228 (597h26m52.283971263s) [noise=true , address=145.79.227.27:4691 , piece_id=AG67HXKZBGSF3JV75G2RC5OXLUSKINQFPOOS2WNSR3R6P5FNPGLQ, seg_pos={0 0}, upload=523705]: Uploading piece... Dec 30 12:55:52.413237172 (597h26m52.284320207s) [noise=true , address=145.79.227.24:4581 , piece_id=XNJDC2HJ6NIEXHNMP33STICUMDD5PQ2Y7UGMWU65ZPNH27R5PVDQ, seg_pos={0 0}, upload=523705]: Uploading piece... Dec 30 12:55:52.413653144 (597h26m52.284736179s) [noise=true , address=145.79.227.21:4475 , piece_id=42PVG5Q7APUS6QHI5JP6H2OLYIJSN5FXFSTDCKSNZVRKYPWVNI6A, seg_pos={0 0}, upload=523705]: Uploading piece... Dec 30 12:55:52.414092440 (597h26m52.285175475s) [noise=true , address=45.91.116.13:4153 , piece_id=J27M26BILJ4V23GMBQGK4OVKICA4TE3LDARCMDFPH7W6RWBGUYKQ, seg_pos={0 0}, upload=523705]: Uploading piece... Dec 30 12:55:52.415787007 (597h26m52.286870042s) [noise=true , address=45.91.116.11:4118 , piece_id=2JG6PCUGTT3K6YUMTWNZQB3OXTDZ7RMAFYHHIJMHLVFDY2E5K75A, seg_pos={0 0}, upload=523705]: Uploading piece... Dec 30 12:55:52.415936882 (597h26m52.287019917s) [noise=true , address=45.91.116.14:4189 , piece_id=X7CQTC447CXK6ITDRX4FZN2DNAWCPAUDO2UYH7O2EDBBF2OS35JA, seg_pos={0 0}, upload=523705]: Uploading piece... Dec 30 12:55:52.416357675 (597h26m52.28744072s) [noise=true , address=77.90.31.10:4519 , piece_id=7Q52TQ7IO2B5RCQESR6HYMYW4CZRO5ZPDJPXEAFEKGDPW7FSWE4Q, seg_pos={0 0}, upload=523705]: Uploading piece... Dec 30 12:55:52.416794676 (597h26m52.287877721s) [noise=true , address=45.91.116.9:4262 , piece_id=NCLXSMOIPWB37ABGU6FUUSNFW5MUZ5PQT4YGT4YGCJ2QXNQ6ML6Q, seg_pos={0 0}, upload=523705]: Uploading piece... Dec 30 12:55:52.417238962 (597h26m52.288321997s) [noise=true , address=145.79.227.47:4267 , piece_id=YPMQM5AXQF3LHHNPVIQWNBUHILPO7ORH7P2DUOKG5LMWMXBNVGGA, seg_pos={0 0}, upload=523705]: Uploading piece... Dec 30 12:55:52.417715961 (597h26m52.288798996s) [noise=true , address=45.91.116.7:4012 , piece_id=NNXDGUVP755B6VQ7L6LFLRYE3KN7KDEMKHEUL7TS3GYPSBXHERHA, seg_pos={0 0}, upload=523705]: Uploading piece... Dec 30 12:55:52.418153584 (597h26m52.289236619s) [noise=true , address=145.79.227.20:4438 , piece_id=LRYZYP6O6RNIIPK2VLTX3A2IL34LBGFFOIINSAMZ3TY5LUNWTCKQ, seg_pos={0 0}, upload=523705]: Uploading piece... Dec 30 12:55:54.018730327 (597h26m53.889813362s) [upload=523705]: Next returned nil segment Dec 30 12:55:54.018743952 (597h26m53.889826997s) [upload=523705]: Waiting for error group managing segments... Dec 30 12:55:54.399128062 (597h26m54.270211097s) [noise=true , address=145.79.227.24:4581 , piece_id=XNJDC2HJ6NIEXHNMP33STICUMDD5PQ2Y7UGMWU65ZPNH27R5PVDQ, seg_pos={0 0}, upload=523705]: Done uploading piece. err: <nil> Dec 30 12:55:54.399875084 (597h26m54.270958129s) [noise=true , address=145.79.227.14:4838 , piece_id=AZX5QHDBV7AXA4FIKQ343QISBB3LH24ZHORG53CKKKGGXDXC4QWQ, seg_pos={0 0}, upload=523705]: Done uploading piece. err: <nil> Dec 30 12:55:54.404232521 (597h26m54.275315556s) [noise=true , address=145.79.227.27:4691 , piece_id=AG67HXKZBGSF3JV75G2RC5OXLUSKINQFPOOS2WNSR3R6P5FNPGLQ, seg_pos={0 0}, upload=523705]: Done uploading piece. err: <nil> Dec 30 12:55:54.424052321 (597h26m54.295135356s) [noise=true , address=145.79.227.20:4438 , piece_id=LRYZYP6O6RNIIPK2VLTX3A2IL34LBGFFOIINSAMZ3TY5LUNWTCKQ, seg_pos={0 0}, upload=523705]: Done uploading piece. err: <nil> Dec 30 12:55:54.433077286 (597h26m54.304160321s) [noise=true , address=145.79.227.21:4475 , piece_id=42PVG5Q7APUS6QHI5JP6H2OLYIJSN5FXFSTDCKSNZVRKYPWVNI6A, seg_pos={0 0}, upload=523705]: Done uploading piece. err: <nil> Dec 30 12:55:54.437149530 (597h26m54.308232575s) [noise=true , address=145.79.227.47:4267 , piece_id=YPMQM5AXQF3LHHNPVIQWNBUHILPO7ORH7P2DUOKG5LMWMXBNVGGA, seg_pos={0 0}, upload=523705]: Done uploading piece. err: <nil> Dec 30 12:55:54.480458237 (597h26m54.351541282s) [noise=true , address=145.79.227.50:4373 , piece_id=BA2SH47KSKE7F5Q4N5HXVNYDZ5FR453LMZHUDBWZZZKYKOIRZXEQ, seg_pos={0 0}, upload=523705]: Done uploading piece. err: <nil> Dec 30 12:55:54.689140229 (597h26m54.560223274s) [noise=true , address=77.90.31.13:4626 , piece_id=TIHHT6FYFWUDFIVQZVQLE5IAL5Q2E3TKHDDIUBZSGFBQKQS4W5GA, seg_pos={0 0}, upload=523705]: Done uploading piece. err: <nil> Dec 30 12:55:54.711163926 (597h26m54.582246961s) [noise=true , address=45.91.116.13:4153 , piece_id=J27M26BILJ4V23GMBQGK4OVKICA4TE3LDARCMDFPH7W6RWBGUYKQ, seg_pos={0 0}, upload=523705]: Done uploading piece. err: <nil> Dec 30 12:55:54.730262152 (597h26m54.601345197s) [noise=true , address=45.91.116.11:4118 , piece_id=2JG6PCUGTT3K6YUMTWNZQB3OXTDZ7RMAFYHHIJMHLVFDY2E5K75A, seg_pos={0 0}, upload=523705]: Done uploading piece. err: <nil> Dec 30 12:55:54.877886751 (597h26m54.748969796s) [noise=true , address=77.90.31.10:4519 , piece_id=7Q52TQ7IO2B5RCQESR6HYMYW4CZRO5ZPDJPXEAFEKGDPW7FSWE4Q, seg_pos={0 0}, upload=523705]: Done uploading piece. err: <nil> Dec 30 12:55:54.889062301 (597h26m54.760145336s) [noise=true , address=45.91.116.8:4048 , piece_id=UXYGO5E3UBXUJKN442OB6CJWQYGW6Y5XUGABC467V2EIZP4KILTA, seg_pos={0 0}, upload=523705]: Done uploading piece. err: <nil> Dec 30 12:55:55.074468032 (597h26m54.945551077s) [noise=true , address=45.91.116.9:4262 , piece_id=NCLXSMOIPWB37ABGU6FUUSNFW5MUZ5PQT4YGT4YGCJ2QXNQ6ML6Q, seg_pos={0 0}, upload=523705]: Done uploading piece. err: <nil> Dec 30 12:55:55.074489473 (597h26m54.945572518s) [seg_pos={0 0}, upload=523705]: Segment reached optimal threshold of 13 pieces. Dec 30 12:55:55.074727766 (597h26m54.945810811s) [noise=true , address=45.91.116.14:4189 , piece_id=X7CQTC447CXK6ITDRX4FZN2DNAWCPAUDO2UYH7O2EDBBF2OS35JA, seg_pos={0 0}, upload=523705]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: write tcp 188.166.225.170:57284->45.91.116.14:4189: use of closed network connection; write tcp 188.166.225.170:57284->45.91.116.14:4189: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57284->45.91.116.14:4189: use of closed network connection Dec 30 12:55:55.074735361 (597h26m54.945818396s) [noise=true , address=45.91.116.14:4189 , piece_id=X7CQTC447CXK6ITDRX4FZN2DNAWCPAUDO2UYH7O2EDBBF2OS35JA, seg_pos={0 0}, upload=523705]: Long tail cancellation due to optimal threshold Dec 30 12:55:55.074903922 (597h26m54.945986967s) [noise=true , address=45.91.116.7:4012 , piece_id=NNXDGUVP755B6VQ7L6LFLRYE3KN7KDEMKHEUL7TS3GYPSBXHERHA, seg_pos={0 0}, upload=523705]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t): protocol: write tcp 188.166.225.170:49534->45.91.116.7:4012: use of closed network connection; write tcp 188.166.225.170:49534->45.91.116.7:4012: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49534->45.91.116.7:4012: use of closed network connection Dec 30 12:55:55.074929440 (597h26m54.946012485s) [noise=true , address=45.91.116.7:4012 , piece_id=NNXDGUVP755B6VQ7L6LFLRYE3KN7KDEMKHEUL7TS3GYPSBXHERHA, seg_pos={0 0}, upload=523705]: Long tail cancellation due to optimal threshold Dec 30 12:55:55.074969206 (597h26m54.946052241s) [noise=true , address=45.91.116.6:4973 , piece_id=OT2DELPCMUI22A444ULTMOO4CYN4VD32E654SR44MDN2WQUATGDA, seg_pos={0 0}, upload=523705]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV): protocol: expected piece hash; context canceled; EOF Dec 30 12:55:55.074975178 (597h26m54.946058213s) [noise=true , address=45.91.116.6:4973 , piece_id=OT2DELPCMUI22A444ULTMOO4CYN4VD32E654SR44MDN2WQUATGDA, seg_pos={0 0}, upload=523705]: Long tail cancellation due to optimal threshold Dec 30 12:55:55.075042837 (597h26m54.946125872s) [seg_pos={0 0}, upload=523705]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 12:55:55.075106016 (597h26m54.946189051s) [upload=523705]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 12:55:55.075110695 (597h26m54.94619373s) [upload=523705]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 12:55:55.365579192 (597h26m55.236662227s) [upload=523705]: Done uploading segments... Dec 30 12:55:55.365601594 (597h26m55.236684639s) [upload=523705]: Upload finished. err: <nil> Duration: 28.381347434s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [..................X.......................X....X............] duration: 17.71507809s trace: https://jaeger.dev.storj.io/trace/7b47a373389cd7c3 FAIL (2025-12-30 13:01:11 UTC) orbiter warning: context deadline exceeded Duration: 17.715090103s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 45 out of 60 is failed (threshold=6) [.XXX.XX.XXX...X.XXXXXXXXXXX.X.XXXXXXXXX.X.XXXXXX.X.XXX.XXXX.] duration: 3.830721569s trace: https://jaeger.dev.storj.io/trace/7bfe418e178b51b1 FAIL (2025-12-30 13:00:58 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.830800348s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [................................X........X...........X..X...] duration: 19.822188341s trace: https://jaeger.dev.storj.io/trace/17a3dfac1fc2c808 FAIL (2025-12-30 13:00:56 UTC) orbiter warning: context deadline exceeded Duration: 19.822205413s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [...................X.......................X....X...........] duration: 17.486281869s trace: https://jaeger.dev.storj.io/trace/7b47a373389cd7c3 FAIL (2025-12-30 13:00:46 UTC) orbiter warning: context deadline exceeded Duration: 17.48629873s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.................................X........X...........X..X..] duration: 23.059003725s trace: https://jaeger.dev.storj.io/trace/17a3dfac1fc2c808 FAIL (2025-12-30 13:00:36 UTC) orbiter warning: context deadline exceeded Duration: 23.059035156s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 7 out of 60 is failed (threshold=3) [.............X.....X.X...X.........X....X........X..........] duration: 27.593743599s trace: https://jaeger.dev.storj.io/trace/1e6433ec4a141391 FAIL (2025-12-30 13:00:25 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== [noise=true , address=77.90.31.11:4583 , piece_id=JEMHBN5CJQI4GTDPRB5RZLBRDUWB46JL7JTORRZXYJTLFK7Z66PA, seg_pos={0 0}, upload=523642]: Uploading piece... Dec 30 12:51:35.352305080 (597h22m35.223388115s) [noise=true , address=145.79.227.44:4222 , piece_id=32QNUAYLXL4AQYBRI6LDICOWQIZMSU3OVN5SWVOCB6XIZY4FHRSA, seg_pos={0 0}, upload=523642]: Uploading piece... Dec 30 12:51:35.353083031 (597h22m35.224166066s) [noise=true , address=145.79.227.34:4925 , piece_id=G46JE4KCDC7FPDEUW2YDTSLI5X66MAMDCRRLDG7EBYF6ENLELLEQ, seg_pos={0 0}, upload=523642]: Uploading piece... Dec 30 12:51:35.353482140 (597h22m35.224565175s) [noise=true , address=145.79.227.33:4897 , piece_id=6SGUOB6IMRF2SPO6WHSKB6XLQENVS5TCJ6RMO6BYIMQ2PANMHK6A, seg_pos={0 0}, upload=523642]: Uploading piece... Dec 30 12:51:35.354315267 (597h22m35.225398302s) [noise=true , address=145.79.227.17:4991 , piece_id=GUPSOAIXZGPRJ7MVZ46464IRG7FBQEFR2RHYDT2GIPAMAKN36YHQ, seg_pos={0 0}, upload=523642]: Uploading piece... Dec 30 12:51:35.354723675 (597h22m35.22580671s) [noise=true , address=145.79.227.2:4119 , piece_id=55AC2GONNJPSGZJ3Y3CYFRLUVKRDMZGLY4HMAPDMEGMN6AJCYWOQ, seg_pos={0 0}, upload=523642]: Uploading piece... Dec 30 12:51:35.355352552 (597h22m35.226435587s) [noise=true , address=64.49.13.169:4716 , piece_id=EIRJU5OSYOF3D3AWCGKKSPFL36THXWUWJWIMBYHVEPI7VB6ATD3Q, seg_pos={0 0}, upload=523642]: Uploading piece... Dec 30 12:51:35.356361994 (597h22m35.227445029s) [noise=true , address=145.79.227.49:4367 , piece_id=6L53GGMES5PBPHPLUCPDUGYKKOQQ64VSZWAGX6EP755TJXT5RH6A, seg_pos={0 0}, upload=523642]: Uploading piece... Dec 30 12:51:35.356578216 (597h22m35.227661261s) [noise=true , address=77.90.31.14:4682 , piece_id=XACXXUOJEEGBXQEJFZYOZKQDX5N2V3LX5NCEQP64YIBZPHK23NQA, seg_pos={0 0}, upload=523642]: Uploading piece... Dec 30 12:51:35.357840370 (597h22m35.228923405s) [noise=true , address=145.79.227.46:4612 , piece_id=VG2ZGVJNHBN2GT72UXK6QA32HYGW6465MEOII6GYSOS7SZYLBBJA, seg_pos={0 0}, upload=523642]: Uploading piece... Dec 30 12:51:35.359039433 (597h22m35.230122468s) [noise=true , address=145.79.227.40:4072 , piece_id=ACK67AVKO5NK7IJKXGL2SLN4GGEKGW47WNTCZCPHIAKIVXYGFHVA, seg_pos={0 0}, upload=523642]: Uploading piece... Dec 30 12:51:35.359333493 (597h22m35.230416538s) [noise=true , address=77.90.31.18:4812 , piece_id=V347IHYDQTVIQ43JTHQJJW2BU3E4VSOSWABCPJPVX2JVCR6J6UNA, seg_pos={0 0}, upload=523642]: Uploading piece... Dec 30 12:51:35.360694154 (597h22m35.231777189s) [noise=true , address=45.91.116.9:4289 , piece_id=O735F2PWWAXGOP54RXURLZ64S7XC4XZZY4OH7RM4N2UOIAA3LJTA, seg_pos={0 0}, upload=523642]: Uploading piece... Dec 30 12:51:35.361764952 (597h22m35.232847987s) [noise=true , address=145.79.227.43:4160 , piece_id=746FBAMI37I2B2UMEKDGOVGNDHN5EXJJH6CMGSCTEEOAVLNDXVXA, seg_pos={0 0}, upload=523642]: Uploading piece... Dec 30 12:51:35.363365710 (597h22m35.234448755s) [noise=true , address=145.79.227.19:4099 , piece_id=DLLN7EUVR2D72XD7BL57AW5JPHVE76PUWF6LWXDEZQH6JZVH3RZQ, seg_pos={0 0}, upload=523642]: Uploading piece... Dec 30 12:51:35.363684648 (597h22m35.234767683s) [noise=true , address=145.79.227.30:4775 , piece_id=JADWMGGTCCSYQMINZV4CLHKEAWREFOS5S4RSY2MFBKC3ULFBD7FA, seg_pos={0 0}, upload=523642]: Uploading piece... Dec 30 12:51:36.749009517 (597h22m36.620092552s) [upload=523642]: Next returned nil segment Dec 30 12:51:36.749023393 (597h22m36.620106428s) [upload=523642]: Waiting for error group managing segments... Dec 30 12:51:37.208596330 (597h22m37.079679376s) [noise=true , address=145.79.227.17:4991 , piece_id=GUPSOAIXZGPRJ7MVZ46464IRG7FBQEFR2RHYDT2GIPAMAKN36YHQ, seg_pos={0 0}, upload=523642]: Done uploading piece. err: <nil> Dec 30 12:51:37.240881687 (597h22m37.111964722s) [noise=true , address=145.79.227.49:4367 , piece_id=6L53GGMES5PBPHPLUCPDUGYKKOQQ64VSZWAGX6EP755TJXT5RH6A, seg_pos={0 0}, upload=523642]: Done uploading piece. err: <nil> Dec 30 12:51:37.294293140 (597h22m37.165376175s) [noise=true , address=145.79.227.2:4119 , piece_id=55AC2GONNJPSGZJ3Y3CYFRLUVKRDMZGLY4HMAPDMEGMN6AJCYWOQ, seg_pos={0 0}, upload=523642]: Done uploading piece. err: <nil> Dec 30 12:51:37.302040142 (597h22m37.173123177s) [noise=true , address=145.79.227.43:4160 , piece_id=746FBAMI37I2B2UMEKDGOVGNDHN5EXJJH6CMGSCTEEOAVLNDXVXA, seg_pos={0 0}, upload=523642]: Done uploading piece. err: <nil> Dec 30 12:51:37.308919160 (597h22m37.180002196s) [noise=true , address=145.79.227.19:4099 , piece_id=DLLN7EUVR2D72XD7BL57AW5JPHVE76PUWF6LWXDEZQH6JZVH3RZQ, seg_pos={0 0}, upload=523642]: Done uploading piece. err: <nil> Dec 30 12:51:37.370286944 (597h22m37.24136999s) [noise=true , address=77.90.31.10:4545 , piece_id=YQV3IPXMIBSO5ZCISFUBGRGHE6FIXLKTLNYKPWR42VNBKNOGPG5Q, seg_pos={0 0}, upload=523642]: Done uploading piece. err: <nil> Dec 30 12:51:37.397564325 (597h22m37.26864737s) [noise=true , address=77.90.31.18:4812 , piece_id=V347IHYDQTVIQ43JTHQJJW2BU3E4VSOSWABCPJPVX2JVCR6J6UNA, seg_pos={0 0}, upload=523642]: Done uploading piece. err: <nil> Dec 30 12:51:37.481372759 (597h22m37.352455794s) [noise=true , address=77.90.31.14:4682 , piece_id=XACXXUOJEEGBXQEJFZYOZKQDX5N2V3LX5NCEQP64YIBZPHK23NQA, seg_pos={0 0}, upload=523642]: Done uploading piece. err: <nil> Dec 30 12:51:37.868609147 (597h22m37.739692192s) [noise=true , address=145.79.227.46:4612 , piece_id=VG2ZGVJNHBN2GT72UXK6QA32HYGW6465MEOII6GYSOS7SZYLBBJA, seg_pos={0 0}, upload=523642]: Done uploading piece. err: <nil> Dec 30 12:51:38.122711215 (597h22m37.993794251s) [noise=true , address=145.79.227.44:4222 , piece_id=32QNUAYLXL4AQYBRI6LDICOWQIZMSU3OVN5SWVOCB6XIZY4FHRSA, seg_pos={0 0}, upload=523642]: Done uploading piece. err: <nil> Dec 30 12:51:38.156332275 (597h22m38.02741531s) [noise=true , address=145.79.227.30:4775 , piece_id=JADWMGGTCCSYQMINZV4CLHKEAWREFOS5S4RSY2MFBKC3ULFBD7FA, seg_pos={0 0}, upload=523642]: Done uploading piece. err: <nil> Dec 30 12:51:38.210830938 (597h22m38.081913983s) [noise=true , address=145.79.227.34:4925 , piece_id=G46JE4KCDC7FPDEUW2YDTSLI5X66MAMDCRRLDG7EBYF6ENLELLEQ, seg_pos={0 0}, upload=523642]: Done uploading piece. err: <nil> Dec 30 12:51:38.318916249 (597h22m38.189999284s) [noise=true , address=145.79.227.33:4897 , piece_id=6SGUOB6IMRF2SPO6WHSKB6XLQENVS5TCJ6RMO6BYIMQ2PANMHK6A, seg_pos={0 0}, upload=523642]: Done uploading piece. err: <nil> Dec 30 12:51:38.528107081 (597h22m38.399190127s) [noise=true , address=193.149.164.143:4694 , piece_id=AOZVLCCSBEYZX3WES7JJVOA2BECEBPII3A33PG5KCSQY6LSZ6Y3A, seg_pos={0 0}, upload=523642]: Done uploading piece. err: <nil> Dec 30 12:51:38.528133793 (597h22m38.399216838s) [seg_pos={0 0}, upload=523642]: Segment reached optimal threshold of 14 pieces. Dec 30 12:51:38.528364652 (597h22m38.399447687s) [noise=true , address=45.91.116.9:4289 , piece_id=O735F2PWWAXGOP54RXURLZ64S7XC4XZZY4OH7RM4N2UOIAA3LJTA, seg_pos={0 0}, upload=523642]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rfBAoVRdhYiRJFWqnamoRAFLxEUsk7tShJEG3jvhKDbcHwipD): protocol: expected piece hash; context canceled; EOF Dec 30 12:51:38.528373429 (597h22m38.399456464s) [noise=true , address=45.91.116.9:4289 , piece_id=O735F2PWWAXGOP54RXURLZ64S7XC4XZZY4OH7RM4N2UOIAA3LJTA, seg_pos={0 0}, upload=523642]: Long tail cancellation due to optimal threshold Dec 30 12:51:38.528672969 (597h22m38.399756014s) [noise=true , address=145.79.227.40:4072 , piece_id=ACK67AVKO5NK7IJKXGL2SLN4GGEKGW47WNTCZCPHIAKIVXYGFHVA, seg_pos={0 0}, upload=523642]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FSQUJNBPY52p2Wkg48DWPN3bdciabNacXcgDEHSwLRrnPG9ne): protocol: expected piece hash; context canceled; EOF Dec 30 12:51:38.528682567 (597h22m38.399765602s) [noise=true , address=145.79.227.40:4072 , piece_id=ACK67AVKO5NK7IJKXGL2SLN4GGEKGW47WNTCZCPHIAKIVXYGFHVA, seg_pos={0 0}, upload=523642]: Long tail cancellation due to optimal threshold Dec 30 12:51:38.528839085 (597h22m38.39992212s) [noise=true , address=77.90.31.11:4583 , piece_id=JEMHBN5CJQI4GTDPRB5RZLBRDUWB46JL7JTORRZXYJTLFK7Z66PA, seg_pos={0 0}, upload=523642]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Ttdx6LHmuAt8JVC7arCDgWjm9eLadjeaEw17F12btoXuSiq9t): protocol: expected piece hash; context canceled; EOF Dec 30 12:51:38.528847421 (597h22m38.399930476s) [noise=true , address=77.90.31.11:4583 , piece_id=JEMHBN5CJQI4GTDPRB5RZLBRDUWB46JL7JTORRZXYJTLFK7Z66PA, seg_pos={0 0}, upload=523642]: Long tail cancellation due to optimal threshold Dec 30 12:51:38.529001655 (597h22m38.4000847s) [noise=true , address=64.49.13.169:4716 , piece_id=EIRJU5OSYOF3D3AWCGKKSPFL36THXWUWJWIMBYHVEPI7VB6ATD3Q, seg_pos={0 0}, upload=523642]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1w21xeA7emhjphgYw2Qyz3uuHgPezuivmVD9KmrKmNZk3TnaGN): protocol: write tcp 188.166.225.170:59670->64.49.13.169:4716: use of closed network connection; write tcp 188.166.225.170:59670->64.49.13.169:4716: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59670->64.49.13.169:4716: use of closed network connection Dec 30 12:51:38.529010362 (597h22m38.400093407s) [noise=true , address=64.49.13.169:4716 , piece_id=EIRJU5OSYOF3D3AWCGKKSPFL36THXWUWJWIMBYHVEPI7VB6ATD3Q, seg_pos={0 0}, upload=523642]: Long tail cancellation due to optimal threshold Dec 30 12:51:38.529029568 (597h22m38.400112603s) [seg_pos={0 0}, upload=523642]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 12:51:38.529107577 (597h22m38.400190612s) [upload=523642]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 12:51:38.529114079 (597h22m38.400197124s) [upload=523642]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 12:51:38.828356312 (597h22m38.699439387s) [upload=523642]: Done uploading segments... Dec 30 12:51:38.828379166 (597h22m38.699462231s) [upload=523642]: Upload finished. err: <nil> Duration: 27.593772425s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [....................X.......................X....X..........] duration: 8.699956652s trace: https://jaeger.dev.storj.io/trace/7b47a373389cd7c3 FAIL (2025-12-30 13:00:17 UTC) orbiter warning: context deadline exceeded Duration: 8.699996527s core uplink us-select-1 us1-placement-20

1
...
214
215
216
217
218
...
7109